You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/05/06 01:34:06 UTC

[GitHub] [spark] Kimahriman opened a new pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Kimahriman opened a new pull request #32448:
URL: https://github.com/apache/spark/pull/32448


   <!--
   Thanks for sending a pull request!  Here are some tips for you:
     1. If this is your first time, please read our contributor guidelines: https://spark.apache.org/contributing.html
     2. Ensure you have added or run the appropriate tests for your PR: https://spark.apache.org/developer-tools.html
     3. If the PR is unfinished, add '[WIP]' in your PR title, e.g., '[WIP][SPARK-XXXX] Your PR title ...'.
     4. Be sure to keep the PR description updated to reflect all changes.
     5. Please write your PR title to summarize what this PR proposes.
     6. If possible, provide a concise example to reproduce the issue for a faster review.
     7. If you want to add a new configuration, please read the guideline first for naming configurations in
        'core/src/main/scala/org/apache/spark/internal/config/ConfigEntry.scala'.
     8. If you want to add or modify an error message, please read the guideline first:
        https://spark.apache.org/error-message-guidelines.html
   -->
   
   ### What changes were proposed in this pull request?
   <!--
   Please clarify what changes you are proposing. The purpose of this section is to outline the changes and how this PR fixes the issue. 
   If possible, please consider writing useful notes for better and faster reviews in your PR. See the examples below.
     1. If you refactor some codes with changing classes, showing the class hierarchy will help reviewers.
     2. If you fix some SQL features, you can provide some references of other DBMSes.
     3. If there is design documentation, please add the link.
     4. If there is a discussion in the mailing list, please add the link.
   -->
   This PR changes the unionByName with null filling logic to use StructType.merge to generate the final merged schema versus sorting fields alphabetically. It removes the need to use UpdateField expressions, and just directly projects new nested structs from each side of the union with the correct schema. This changes the union'd schema from being alphabetically sorted previously to now "left dominant", where the fields from the left side of the union are included and then the missing ones from the right are added in the same order found originally.
   
   I did have to add a resolver to the StructType merging to handle case insensitivity. I'm not sure if it was a bug or a feature that it wasn't doing that before, so definitely interested in feedback on that.
   
   
   ### Why are the changes needed?
   <!--
   Please clarify why the changes are needed. For instance,
     1. If you propose a new API, clarify the use case for a new API.
     2. If you fix a bug, you can clarify why it is a bug.
   -->
   Certain nested structs would cause unionByName with null filling to error out due to part of the logic for rewriting the expression tree to sort the structs.
   
   ### Does this PR introduce _any_ user-facing change?
   <!--
   Note that it means *any* user-facing change including all aspects such as the documentation fix.
   If yes, please clarify the previous behavior and the change this PR proposes - provide the console output, description and/or an example to show the behavior difference if possible.
   If possible, please also clarify if this is a user-facing change compared to the released Spark versions or within the unreleased branches such as master.
   If no, write 'No'.
   -->
   Shouldn't other than fixing certain cases that caused errors. I don't know if adding the resolver to the StructType merging has any unintended side effects, so definitely would like some thoughts on that. Also the order of the StructFields is slightly different now, though that shouldn't have too much of an effect.
   
   ### How was this patch tested?
   <!--
   If tests were added, say they were added here. Please make sure to add some test cases that check the changes thoroughly including negative and positive cases if possible.
   If it was tested in a way different from regular unit tests, please clarify how you tested step by step, ideally copy and paste-able, so that other reviewers can test and check, and descendants can verify in the future.
   If tests were not added, please describe why they were not added and/or why it was difficult to add.
   -->
   Updated existing tests based on the new StructField ordering and added a new test for the case that was broken originally.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627020424



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -555,32 +555,31 @@ object StructType extends AbstractDataType {
       case _ => dt
     }
 
-  private[sql] def merge(left: DataType, right: DataType): DataType =
+  private[sql] def merge(resolver: Resolver)(left: DataType, right: DataType): DataType =

Review comment:
       I can add tests for this if we want to keep this behavior




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849345266


   **[Test build #138998 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138998/testReport)** for PR 32448 at commit [`b5b56bd`](https://github.com/apache/spark/commit/b5b56bd4be402cc0bd804624c8b0c723053d78a5).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633698385



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       The union throws an analysis exception because the right side gets corrupted. It's the same example I have in the issue: https://issues.apache.org/jira/browse/SPARK-35290




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840891914


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43049/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629786302



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       Following on to that, how is it determined when things are backported to previous releases (3.1 in this case) versus saved for the next minor release? Is that up to submitters or do maintainers make that call? There's a little bit of a "behavior change" here, though it's mostly a bug fix. Similar with https://github.com/apache/spark/pull/32338, where it's a bug fix that could be useful in a 3.1 patch release.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834126745


   ok to test


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840339407


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138481/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-833167741






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849246809


   **[Test build #138996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138996/testReport)** for PR 32448 at commit [`1fe3c13`](https://github.com/apache/spark/commit/1fe3c1356af65a5a7385aeb31ab710758f1a52b2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849248384


   **[Test build #138998 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138998/testReport)** for PR 32448 at commit [`b5b56bd`](https://github.com/apache/spark/commit/b5b56bd4be402cc0bd804624c8b0c723053d78a5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-833167741


   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r650478478



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
##########
@@ -150,95 +150,36 @@ class StructTypeSuite extends SparkFunSuite with SQLHelper {
     assert(fromDDL(interval).toDDL === interval)
   }
 
-  test("find missing (nested) fields") {
-    val schema = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
+  test("SPARK-35290: Struct merging case insensitive") {
+    val schema1 = StructType.fromDDL("a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING>")
+    val schema2 = StructType.fromDDL("A2 STRING, a3 DOUBLE, nested STRUCT<B2: STRING, b3: DOUBLE>")
     val resolver = SQLConf.get.resolver
 
-    val source1 = StructType.fromDDL("c1 INT")
-    val missing1 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source1, schema, resolver)
-      .exists(_.sameType(missing1)))
+    assert(schema1.merge(schema2, resolver).sameType(StructType.fromDDL(
+      "a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING, b3: DOUBLE>, a3 DOUBLE"
+    )))
 
-    val source2 = StructType.fromDDL("c1 INT, c3 STRING")
-    val missing2 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source2, schema, resolver)
-      .exists(_.sameType(missing2)))
-
-    val source3 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT>")
-    val missing3 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source3, schema, resolver)
-      .exists(_.sameType(missing3)))
-
-    val source4 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c6: INT>>")
-    val missing4 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT>>")
-    assert(StructType.findMissingFields(source4, schema, resolver)
-      .exists(_.sameType(missing4)))
+    assert(schema2.merge(schema1, resolver).sameType(StructType.fromDDL(
+      "a2 STRING, a3 DOUBLE, nested STRUCT<b2: STRING, b3: DOUBLE, b1: INT>, a1 INT"

Review comment:
       When `schema2` mergs `schema1`, don't we keep its original case? E.g. `"A2 STRING, a3 DOUBLE, nested STRUCT<B2: STRING, b3: DOUBLE, b1: INT>, a1 INT"`

##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
##########
@@ -150,95 +150,36 @@ class StructTypeSuite extends SparkFunSuite with SQLHelper {
     assert(fromDDL(interval).toDDL === interval)
   }
 
-  test("find missing (nested) fields") {
-    val schema = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
+  test("SPARK-35290: Struct merging case insensitive") {
+    val schema1 = StructType.fromDDL("a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING>")
+    val schema2 = StructType.fromDDL("A2 STRING, a3 DOUBLE, nested STRUCT<B2: STRING, b3: DOUBLE>")
     val resolver = SQLConf.get.resolver
 
-    val source1 = StructType.fromDDL("c1 INT")
-    val missing1 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source1, schema, resolver)
-      .exists(_.sameType(missing1)))
+    assert(schema1.merge(schema2, resolver).sameType(StructType.fromDDL(
+      "a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING, b3: DOUBLE>, a3 DOUBLE"
+    )))
 
-    val source2 = StructType.fromDDL("c1 INT, c3 STRING")
-    val missing2 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source2, schema, resolver)
-      .exists(_.sameType(missing2)))
-
-    val source3 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT>")
-    val missing3 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source3, schema, resolver)
-      .exists(_.sameType(missing3)))
-
-    val source4 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c6: INT>>")
-    val missing4 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT>>")
-    assert(StructType.findMissingFields(source4, schema, resolver)
-      .exists(_.sameType(missing4)))
+    assert(schema2.merge(schema1, resolver).sameType(StructType.fromDDL(
+      "a2 STRING, a3 DOUBLE, nested STRUCT<b2: STRING, b3: DOUBLE, b1: INT>, a1 INT"

Review comment:
       When `schema2` merges `schema1`, don't we keep its original case? E.g. `"A2 STRING, a3 DOUBLE, nested STRUCT<B2: STRING, b3: DOUBLE, b1: INT>, a1 INT"`

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -181,7 +100,8 @@ object ResolveUnion extends Rule[LogicalPlan] {
             // like that. We will sort columns in the struct expression to make sure two sides of
             // union have consistent schema.
             aliased += foundAttr
-            Alias(addFields(foundAttr, target), foundAttr.name)()
+            val targetType = target.merge(source, conf.resolver)

Review comment:
       BTW, `merge` will throw an exception if two schemas conflict. I recall that union of conflicting schemas doesn't fail in `ResolveUnion`, but in `CheckAnalysis`. Could we follow original behavior?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840339407


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138481/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631515982



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -555,32 +555,31 @@ object StructType extends AbstractDataType {
       case _ => dt
     }
 
-  private[sql] def merge(left: DataType, right: DataType): DataType =
+  private[sql] def merge(resolver: Resolver)(left: DataType, right: DataType): DataType =

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633717463



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I see. From the JIRA, seems the problem is `b:struct<ba:string:bb:string>` becomes `b:struct<aa:string,bb:string>` due to some reasons. Can we fix the bug surgically without changing how the fields are ordered? Or do we have other reasons to change the behavior?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-843851065


   The code change itself looks good. It's better to clarify why there was a failure before. Sorting the nested fields shouldn't lead to a bug that `b:struct<ba:string:bb:string>` becomes `b:struct<aa:string,bb:string>`. 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634486694



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I think sorting is reasonable as well. `df1.unionByName(df2)` can always have the same schema of `df2.unionByName(df1)`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849350972


   **[Test build #138996 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138996/testReport)** for PR 32448 at commit [`1fe3c13`](https://github.com/apache/spark/commit/1fe3c1356af65a5a7385aeb31ab710758f1a52b2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861761935


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139825/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861851212


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840729216


   #32533 is merged, can you rebase to get the fix?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r635011079



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I originally planed to work on to get rid of the sorting behavior but have no time on it. I think it is more consistent with top-level union by name behavior. Currently this approach looks much simpler than `withField` approach and can easily get rid of the sorting behavior. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840237394






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-852785583


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43723/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631516355



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -752,8 +786,50 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1: DataFrame = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2: DataFrame = spark.createDataFrame(

Review comment:
       Done, and took a few others out as well




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834599260


   Thanks for working on this! It looks like a better approach. Let me take a closer look in next few days.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834128258


   **[Test build #138240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138240/testReport)** for PR 32448 at commit [`3c0d3d0`](https://github.com/apache/spark/commit/3c0d3d0af7d89d28dbe1db7b0c1b8e9dc8c090a5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629344269



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       You want a 3.1 -> 3.2 migration message saying fields are no longer sorted but instead kept in order with new fields added to the end?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629773979



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       I just copied this from the `findNestedField` definition below in this class. I could also just remove the default, I think a test in the DataTypeSuite is the only thing not passing a value. Don't have a strong preference either way




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848389402


   **[Test build #138950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138950/testReport)** for PR 32448 at commit [`df38171`](https://github.com/apache/spark/commit/df38171b371f41d4470addbc381d45fdb4147e4b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840885868






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860056707






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634467977



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       It was "designed" because it was thought to be the only way. The author said this might be a better approach. I can see if I can remember what that update was, never thoroughly tested it




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848444120


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43470/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834128258


   **[Test build #138240 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138240/testReport)** for PR 32448 at commit [`3c0d3d0`](https://github.com/apache/spark/commit/3c0d3d0af7d89d28dbe1db7b0c1b8e9dc8c090a5).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834298754


   **[Test build #138240 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138240/testReport)** for PR 32448 at commit [`3c0d3d0`](https://github.com/apache/spark/commit/3c0d3d0af7d89d28dbe1db7b0c1b8e9dc8c090a5).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r632017258



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")
+      checkAnswer(unionDf,
+        Row(0, Row(0, 1, Row(1, "2", null, null))) ::
+          Row(1, Row(1, 2, Row(null, null, 2, 3L))) :: Nil)
 
       unionDf = df2.unionByName(df1, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L, null))) ::
-          Row(0, Row(0, 1, Row(null, 1, null, "2"))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT, `c`: STRING>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null, null))) ::
+          Row(0, Row(0, 1, Row(null, null, 1, "2"))) :: Nil)
 
       val df3 = Seq((2, UnionClass1b(2, 3L, UnionClass3(4, 5L)))).toDF("id", "a")
       unionDf = df2.unionByName(df3, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L))) ::
-          Row(2, Row(2, 3, Row(null, 4, 5L))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null))) ::
+          Row(2, Row(2, 3, Row(null, 5L, 4))) :: Nil)
     }
   }
 
+  test("SPARK-32376: Make unionByName null-filling behavior work with struct columns" +
+      " - case-insensitive cases") {

Review comment:
       From PR description
   ```
   Certain nested structs would cause unionByName with null filling to error out due to
   part of the logic for rewriting the expression tree to sort the structs.
   ```
   
   Does this test fail before this PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-865945873


   Any thoughts? I think it definitely makes sense to split this into two PRs now since there's two separate bugs being fixed. I could either backout all of the StructType.merge changes in this PR to only have the unionByName fix (that doesn't require StructType.merge anymore and I'm annoyed I didn't figure that out sooner), or close this PR and start fresh with two new PRs (and a new JIRA for the separate bug that the StructType.merge update fixes)


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r640228110



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")
+      checkAnswer(unionDf,
+        Row(0, Row(0, 1, Row(1, "2", null, null))) ::
+          Row(1, Row(1, 2, Row(null, null, 2, 3L))) :: Nil)
 
       unionDf = df2.unionByName(df1, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L, null))) ::
-          Row(0, Row(0, 1, Row(null, 1, null, "2"))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT, `c`: STRING>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null, null))) ::
+          Row(0, Row(0, 1, Row(null, null, 1, "2"))) :: Nil)
 
       val df3 = Seq((2, UnionClass1b(2, 3L, UnionClass3(4, 5L)))).toDF("id", "a")
       unionDf = df2.unionByName(df3, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L))) ::
-          Row(2, Row(2, 3, Row(null, 4, 5L))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null))) ::
+          Row(2, Row(2, 3, Row(null, 5L, 4))) :: Nil)
     }
   }
 
+  test("SPARK-32376: Make unionByName null-filling behavior work with struct columns" +
+      " - case-insensitive cases") {

Review comment:
       I removed this test since it's not really testing anything new, just added it to understand what was happening




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849246809


   **[Test build #138996 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138996/testReport)** for PR 32448 at commit [`1fe3c13`](https://github.com/apache/spark/commit/1fe3c1356af65a5a7385aeb31ab710758f1a52b2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848434576


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43470/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840240209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43001/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848389402


   **[Test build #138950 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138950/testReport)** for PR 32448 at commit [`df38171`](https://github.com/apache/spark/commit/df38171b371f41d4470addbc381d45fdb4147e4b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861761935


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139825/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848444120


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43470/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840240209


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43001/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834153023






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840970187


   **[Test build #138528 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138528/testReport)** for PR 32448 at commit [`b011c69`](https://github.com/apache/spark/commit/b011c696baaa8c84caf877961a37c2b0a61edca4).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861794437


   **[Test build #139827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139827/testReport)** for PR 32448 at commit [`5762ddc`](https://github.com/apache/spark/commit/5762ddc83e96b640ffdd4125c182646bf9fb03ff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860328810


   So @viirya's comment made me realize that `StructType.merge` isn't quite the right solution since it immediately fails on exact type mismatch and can't handle similar types like float/double. I updated things locally to not use it anymore and just resolve things based on name and let the types figure themselves out later like other unions.
   
   Separately, the `StructType.merge` not considering case sensitivity is still a bug that can crop up with unions (not even just unionByName with null filling).
   
   So I can either just push the update I have here, or I can close this and open two separate PRs to address each individually.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860056419






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631997644



##########
File path: docs/sql-migration-guide.md
##########
@@ -87,6 +87,8 @@ license: |
 
   - In Spark 3.2, Spark supports `DayTimeIntervalType` and `YearMonthIntervalType` as inputs and outputs of `TRANSFORM` clause in Hive `SERDE` mode, the behavior is different between Hive `SERDE` mode and `ROW FORMAT DELIMITED` mode when these two types are used as inputs. In Hive `SERDE` mode, `DayTimeIntervalType` column is converted to `HiveIntervalDayTime`, its string format is `[-]?d h:m:s.n`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL '[-]?d h:m:s.n' DAY TO TIME`. In Hive `SERDE` mode, `YearMonthIntervalType` column is converted to `HiveIntervalYearMonth`, its string format is `[-]?y-m`, but in `ROW FORMAT DELIMITED` mode the format is `INTERVAL '[-]?y-m' YEAR TO MONTH`.
 
+  - In Spark 3.2, unionByName with allowMissingColumns set to true will add missing nested fields to the end of structs. In Spark 3.1, nested struct fields are sorted alphabetically.

Review comment:
       Let's be specific, `Dataset.unionByName`




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849367458






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman closed pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman closed pull request #32448:
URL: https://github.com/apache/spark/pull/32448


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849249974






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634437524



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       Can you open a PR for that tweak? If possible it's better to fix a bug without changing the designed behavior.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629623996



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {

Review comment:
       Let's also update the method comment.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629091515



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       is this an expected behavior change? and why do we prefer to new behavior?

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       is this an expected behavior change? and why do we prefer the new behavior?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-852759226


   Can one of the admins verify this patch?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-843985070


   > > The code change itself looks good. It's better to clarify why there was a failure before. Sorting the nested fields shouldn't lead to a bug that `b:struct<ba:string:bb:string>` becomes `b:struct<aa:string,bb:string>`.
   > 
   > I'm not sure about it. @Kimahriman Do you have idea? If not, I may take some time debugging it.
   
   It had something to do with `sortStructFieldsInWithFields`. Changing the order of the fields in a `CreateNamedStruct` corrupted the parent or children expressions in the tree. If you change the `transformUp` to `transformDown`, you get the schemas correctly resolve but they actually have the wrong value in the field (i.e. the value of a.aa is actually in column b.ba). But I can't say I fully understand it


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849291962






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848475808


   **[Test build #138950 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138950/testReport)** for PR 32448 at commit [`df38171`](https://github.com/apache/spark/commit/df38171b371f41d4470addbc381d45fdb4147e4b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861757511


   **[Test build #139825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139825/testReport)** for PR 32448 at commit [`4a14101`](https://github.com/apache/spark/commit/4a141017d4c641d8b4b1a0e5844446d093989976).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633698385



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       The union throws an analysis exception because the right side gets corrupted. It's the same example I have in the issue: https://issues.apache.org/jira/browse/SPARK-35290

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I found one tweak that seemed to fix the issue but made the deeply nested case a lot slower to resolve. I went down this route because it sounded like this would have been the ideal behavior originally but sorting seemed like the only option. And it simplifies things a little bit and uses existing struct merging functionality.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861761897


   **[Test build #139825 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139825/testReport)** for PR 32448 at commit [`4a14101`](https://github.com/apache/spark/commit/4a141017d4c641d8b4b1a0e5844446d093989976).
    * This patch **fails to build**.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840499057


   GitHub Test failure looks due to https://github.com/apache/spark/pull/32533


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840218983


   **[Test build #138481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138481/testReport)** for PR 32448 at commit [`93b47d3`](https://github.com/apache/spark/commit/93b47d3f190369afdf5a2a5ae0ec0c6054b56c1b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627020497



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/parquet/ParquetFileFormat.scala
##########
@@ -429,7 +429,7 @@ object ParquetFileFormat extends Logging {
     }
 
     finalSchemas.reduceOption { (left, right) =>
-      try left.merge(right) catch { case e: Throwable =>
+      try left.merge(right, sparkSession.sessionState.conf.resolver) catch { case e: Throwable =>

Review comment:
       Also don't know enough about this code to know what the impact is




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848476906


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138950/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861924131


   **[Test build #139827 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139827/testReport)** for PR 32448 at commit [`5762ddc`](https://github.com/apache/spark/commit/5762ddc83e96b640ffdd4125c182646bf9fb03ff).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631516011



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -181,7 +100,12 @@ object ResolveUnion extends Rule[LogicalPlan] {
             // like that. We will sort columns in the struct expression to make sure two sides of
             // union have consistent schema.
             aliased += foundAttr
-            Alias(addFields(foundAttr, target), foundAttr.name)()
+            val targetType = try {
+              target.merge(source, conf.resolver)
+            } catch { case e: Throwable =>
+              throw QueryExecutionErrors.failedToMergeIncompatibleSchemasError(target, source, e)

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849291962






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633682592



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       Can you clarify which step fails? I'm reading this test but can't think of how it can fail without this PR.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r632017258



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")
+      checkAnswer(unionDf,
+        Row(0, Row(0, 1, Row(1, "2", null, null))) ::
+          Row(1, Row(1, 2, Row(null, null, 2, 3L))) :: Nil)
 
       unionDf = df2.unionByName(df1, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L, null))) ::
-          Row(0, Row(0, 1, Row(null, 1, null, "2"))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT, `c`: STRING>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null, null))) ::
+          Row(0, Row(0, 1, Row(null, null, 1, "2"))) :: Nil)
 
       val df3 = Seq((2, UnionClass1b(2, 3L, UnionClass3(4, 5L)))).toDF("id", "a")
       unionDf = df2.unionByName(df3, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L))) ::
-          Row(2, Row(2, 3, Row(null, 4, 5L))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null))) ::
+          Row(2, Row(2, 3, Row(null, 5L, 4))) :: Nil)
     }
   }
 
+  test("SPARK-32376: Make unionByName null-filling behavior work with struct columns" +
+      " - case-insensitive cases") {

Review comment:
       From PR description
   ```
   Certain nested structs would cause unionByName with null filling to error out due to part of the logic for rewriting the expression tree to sort the structs.
   ```
   
   Does this test fail before this PR?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849267443


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43515/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634518604



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       It's just weird because it's different behavior than every other part of spark. `withColumn` and `withField` both append to the end. And currently before this PR and if the I applied the tweak `df1.unionByname(df2, True)` would not always have the same schema as `df2.unionByName(df1, True)`. It only sorts nested struct fields, not the top level attributes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627019910



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
##########
@@ -38,7 +38,7 @@ object SchemaPruning extends SQLConfHelper {
     // original schema
     val mergedSchema = requestedRootFields
       .map { root: RootField => StructType(Array(root.field)) }
-      .reduceLeft(_ merge _)
+      .reduceLeft((left, right) => left.merge(right, resolver))

Review comment:
       Don't know enough about this code to know if this could cause any issues




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633725182



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I found one tweak that seemed to fix the issue but made the deeply nested case a lot slower to resolve. I went down this route because it sounded like this would have been the ideal behavior originally but sorting seemed like the only option. And it simplifies things a little bit and uses existing struct merging functionality.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849249974


   **[Test build #139000 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139000/testReport)** for PR 32448 at commit [`ba99753`](https://github.com/apache/spark/commit/ba997539146f4b0bb53d4b62e6b097361db35e77).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860056707






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-852790313


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43723/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-837218278


   Sounds good, wanted to make sure things looked sane before cleaning things up. I'll try and get those comments finished up in the next day or so.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629618659



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/expressions/SchemaPruning.scala
##########
@@ -38,7 +38,7 @@ object SchemaPruning extends SQLConfHelper {
     // original schema
     val mergedSchema = requestedRootFields
       .map { root: RootField => StructType(Array(root.field)) }
-      .reduceLeft(_ merge _)
+      .reduceLeft((left, right) => left.merge(right, resolver))

Review comment:
       This merges some required fields at root level as a single `StructType`. Looks okay.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-866011243


   I'm +1 to open 2 fresh PRs, thanks!


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834311875


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138240/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-852759226






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840218983


   **[Test build #138481 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138481/testReport)** for PR 32448 at commit [`93b47d3`](https://github.com/apache/spark/commit/93b47d3f190369afdf5a2a5ae0ec0c6054b56c1b).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631516151



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       Done

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {
     assert(col.dataType.isInstanceOf[StructType], "Only support StructType.")
 
     val resolver = conf.resolver
-    val missingFieldsOpt =
-      StructType.findMissingFields(col.dataType.asInstanceOf[StructType], target, resolver)
-
-    // We need to sort columns in result, because we might add another column in other side.
-    // E.g., we want to union two structs "a int, b long" and "a int, c string".
-    // If we don't sort, we will have "a int, b long, c string" and
-    // "a int, c string, b long", which are not compatible.
-    if (missingFieldsOpt.isEmpty) {
-      sortStructFields(col)
-    } else {
-      missingFieldsOpt.map { s =>
-        val struct = addFieldsInto(col, s.fields)
-        // Combines `WithFields`s to reduce expression tree.
-        val reducedStruct = struct.transformUp(OptimizeUpdateFields.optimizeUpdateFields)
-        val sorted = sortStructFieldsInWithFields(reducedStruct)
-        sorted
-      }.get
-    }
-  }
-
-  /**
-   * Adds missing fields recursively into given `col` expression. The missing fields are given
-   * in `fields`. For example, given `col` as "z struct<z:int, y:int>, x int", and `fields` is
-   * "z struct<w:long>, w string". This method will add a nested `z.w` field and a top-level
-   * `w` field to `col` and fill null values for them. Note that because we might also add missing
-   * fields at other side of Union, we must make sure corresponding attributes at two sides have
-   * same field order in structs, so when we adding missing fields, we will sort the fields based on
-   * field names. So the data type of returned expression will be
-   * "w string, x int, z struct<w:long, y:int, z:int>".
-   */
-  private def addFieldsInto(
-      col: Expression,
-      fields: Seq[StructField]): Expression = {
-    fields.foldLeft(col) { case (currCol, field) =>
-      field.dataType match {
-        case st: StructType =>
-          val resolver = conf.resolver
-          val colField = currCol.dataType.asInstanceOf[StructType]
-            .find(f => resolver(f.name, field.name))
-          if (colField.isEmpty) {
-            // The whole struct is missing. Add a null.
-            UpdateFields(currCol, field.name, Literal(null, st))
-          } else {
-            UpdateFields(currCol, field.name,
-              addFieldsInto(ExtractValue(currCol, Literal(field.name), resolver), st.fields))
-          }
-        case dt =>
-          UpdateFields(currCol, field.name, Literal(null, dt))
+    val colType = col.dataType.asInstanceOf[StructType]
+    val newStructFields = expectedFields.flatMap(expectedField => {

Review comment:
       Done

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {
     assert(col.dataType.isInstanceOf[StructType], "Only support StructType.")
 
     val resolver = conf.resolver
-    val missingFieldsOpt =
-      StructType.findMissingFields(col.dataType.asInstanceOf[StructType], target, resolver)
-
-    // We need to sort columns in result, because we might add another column in other side.
-    // E.g., we want to union two structs "a int, b long" and "a int, c string".
-    // If we don't sort, we will have "a int, b long, c string" and
-    // "a int, c string, b long", which are not compatible.
-    if (missingFieldsOpt.isEmpty) {
-      sortStructFields(col)
-    } else {
-      missingFieldsOpt.map { s =>
-        val struct = addFieldsInto(col, s.fields)
-        // Combines `WithFields`s to reduce expression tree.
-        val reducedStruct = struct.transformUp(OptimizeUpdateFields.optimizeUpdateFields)
-        val sorted = sortStructFieldsInWithFields(reducedStruct)
-        sorted
-      }.get
-    }
-  }
-
-  /**
-   * Adds missing fields recursively into given `col` expression. The missing fields are given
-   * in `fields`. For example, given `col` as "z struct<z:int, y:int>, x int", and `fields` is
-   * "z struct<w:long>, w string". This method will add a nested `z.w` field and a top-level
-   * `w` field to `col` and fill null values for them. Note that because we might also add missing
-   * fields at other side of Union, we must make sure corresponding attributes at two sides have
-   * same field order in structs, so when we adding missing fields, we will sort the fields based on
-   * field names. So the data type of returned expression will be
-   * "w string, x int, z struct<w:long, y:int, z:int>".
-   */
-  private def addFieldsInto(
-      col: Expression,
-      fields: Seq[StructField]): Expression = {
-    fields.foldLeft(col) { case (currCol, field) =>
-      field.dataType match {
-        case st: StructType =>
-          val resolver = conf.resolver
-          val colField = currCol.dataType.asInstanceOf[StructType]
-            .find(f => resolver(f.name, field.name))
-          if (colField.isEmpty) {
-            // The whole struct is missing. Add a null.
-            UpdateFields(currCol, field.name, Literal(null, st))
-          } else {
-            UpdateFields(currCol, field.name,
-              addFieldsInto(ExtractValue(currCol, Literal(field.name), resolver), st.fields))
-          }
-        case dt =>
-          UpdateFields(currCol, field.name, Literal(null, dt))
+    val colType = col.dataType.asInstanceOf[StructType]
+    val newStructFields = expectedFields.flatMap(expectedField => {
+      val currentField = colType.fields.find(f => resolver(f.name, expectedField.name))
+
+      val newExpression = (currentField, expectedField.dataType) match {
+        case (Some(cf), expectedType: StructType) if cf.dataType.isInstanceOf[StructType] =>
+            val extractedValue = ExtractValue(col, Literal(cf.name), resolver)
+            val combinedStruct = addFields(extractedValue, expectedType.fields)
+            if (extractedValue.nullable) {
+              If(IsNull(extractedValue),
+                Literal(null, combinedStruct.dataType),
+                combinedStruct)
+            } else {
+              combinedStruct
+            }
+        case (Some(cf), _) =>
+          ExtractValue(col, Literal(cf.name), resolver)
+        case (_, expectedType) =>

Review comment:
       Done




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-866751183


   Closing in favor of https://github.com/apache/spark/pull/33040


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-837634074


   Should I remove the findMissingFields function that was added for the original method?


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627317144



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaMergeUtils.scala
##########
@@ -80,7 +80,7 @@ object SchemaMergeUtils extends Logging {
             var mergedSchema = schemas.head
             schemas.tail.foreach { schema =>
               try {
-                mergedSchema = mergedSchema.merge(schema)
+                mergedSchema = mergedSchema.merge(schema, sparkSession.sessionState.conf.resolver)

Review comment:
       Trying to pull the resolver out of session state breaks a lot of the tests. I can revert most of these to not use the resolver and maintain the same behavior for now, but wanted to at least bring up the discussion if it makes sense to do resolver-aware schema merging here. Also, is there some other way you can/supposed to get a resolver? Or is it just a weird test thing?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629786827



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       Yeah the default was just to keep the current (wrong?) behavior. I'll just remove it and add it in the one test that is missing it (and new tests I'll add)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849280766


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43518/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r650522493



##########
File path: sql/catalyst/src/test/scala/org/apache/spark/sql/types/StructTypeSuite.scala
##########
@@ -150,95 +150,36 @@ class StructTypeSuite extends SparkFunSuite with SQLHelper {
     assert(fromDDL(interval).toDDL === interval)
   }
 
-  test("find missing (nested) fields") {
-    val schema = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
+  test("SPARK-35290: Struct merging case insensitive") {
+    val schema1 = StructType.fromDDL("a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING>")
+    val schema2 = StructType.fromDDL("A2 STRING, a3 DOUBLE, nested STRUCT<B2: STRING, b3: DOUBLE>")
     val resolver = SQLConf.get.resolver
 
-    val source1 = StructType.fromDDL("c1 INT")
-    val missing1 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source1, schema, resolver)
-      .exists(_.sameType(missing1)))
+    assert(schema1.merge(schema2, resolver).sameType(StructType.fromDDL(
+      "a1 INT, a2 STRING, nested STRUCT<b1: INT, b2: STRING, b3: DOUBLE>, a3 DOUBLE"
+    )))
 
-    val source2 = StructType.fromDDL("c1 INT, c3 STRING")
-    val missing2 = StructType.fromDDL("c2 STRUCT<c3: INT, c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source2, schema, resolver)
-      .exists(_.sameType(missing2)))
-
-    val source3 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT>")
-    val missing3 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT, c6: INT>>")
-    assert(StructType.findMissingFields(source3, schema, resolver)
-      .exists(_.sameType(missing3)))
-
-    val source4 = StructType.fromDDL("c1 INT, c2 STRUCT<c3: INT, c4: STRUCT<c6: INT>>")
-    val missing4 = StructType.fromDDL("c2 STRUCT<c4: STRUCT<c5: INT>>")
-    assert(StructType.findMissingFields(source4, schema, resolver)
-      .exists(_.sameType(missing4)))
+    assert(schema2.merge(schema1, resolver).sameType(StructType.fromDDL(
+      "a2 STRING, a3 DOUBLE, nested STRUCT<b2: STRING, b3: DOUBLE, b1: INT>, a1 INT"

Review comment:
       Ah yeah it does, `sameType` is just doing a case insensitive comparison so it didn't matter that my manual type was wrong. I'll update to `===` instead and that seems to fix the check

##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -181,7 +100,8 @@ object ResolveUnion extends Rule[LogicalPlan] {
             // like that. We will sort columns in the struct expression to make sure two sides of
             // union have consistent schema.
             aliased += foundAttr
-            Alias(addFields(foundAttr, target), foundAttr.name)()
+            val targetType = target.merge(source, conf.resolver)

Review comment:
       Hmmm good question, I'm not sure exactly how that would work without adding extra logic to StructType.merge to ignore conflicts. And now that you bring that up I'm starting to think using StructType.merge isn't the best method since it does care about DataType. I just noticed it doesn't handle similar types, so you get errors if you try to merge a float and a double, whereas the normal union just handles that. I might try to rework this again to not use the StructType.merge after all... 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627020670



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -181,7 +100,12 @@ object ResolveUnion extends Rule[LogicalPlan] {
             // like that. We will sort columns in the struct expression to make sure two sides of
             // union have consistent schema.
             aliased += foundAttr
-            Alias(addFields(foundAttr, target), foundAttr.name)()
+            val targetType = try {
+              target.merge(source, conf.resolver)
+            } catch { case e: Throwable =>
+              throw QueryExecutionErrors.failedToMergeIncompatibleSchemasError(target, source, e)

Review comment:
       Should I catch this and re-throw this error or just let the original SparkException be thrown for bad merges?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-844334849


   > It had something to do with `sortStructFieldsInWithFields`. Changing the order of the fields in a `CreateNamedStruct` corrupted the parent or children expressions in the tree. If you change the `transformUp` to `transformDown`, you get the schemas correctly resolve but they actually have the wrong value in the field (i.e. the value of a.aa is actually in column b.ba). But I can't say I fully understand it
   
   Yea, yesterday I also found it was happened during `sortStructFieldsInWithFields` after some debugging. Still need some time to look what's going wrong with it.
   
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861865124


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861757511


   **[Test build #139825 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139825/testReport)** for PR 32448 at commit [`4a14101`](https://github.com/apache/spark/commit/4a141017d4c641d8b4b1a0e5844446d093989976).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634588583



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I guess you can look at it that way. It was the most effective way to fix the bug and simplify the code in the process (and not introduce a potential performance issue)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629139604



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       Yeah, the nested fields don't necessarily have to be sorted. The behaviour should be same or at least similar with the outermost schema. Sorting is sort of unexpected I think.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834311875


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138240/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840980471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138528/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861794437


   **[Test build #139827 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139827/testReport)** for PR 32448 at commit [`5762ddc`](https://github.com/apache/spark/commit/5762ddc83e96b640ffdd4125c182646bf9fb03ff).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848407506


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43470/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849268672


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43518/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861813646


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44353/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r627317144



##########
File path: sql/core/src/main/scala/org/apache/spark/sql/execution/datasources/SchemaMergeUtils.scala
##########
@@ -80,7 +80,7 @@ object SchemaMergeUtils extends Logging {
             var mergedSchema = schemas.head
             schemas.tail.foreach { schema =>
               try {
-                mergedSchema = mergedSchema.merge(schema)
+                mergedSchema = mergedSchema.merge(schema, sparkSession.sessionState.conf.resolver)

Review comment:
       Trying to pull the resolver out of session state breaks a lot of the tests. I can revert most of these to not use the resolver and maintain the same behavior for now, but wanted to at least bring up the discussion if it makes sense to do resolver-aware schema merging here. Also, is there some other way you can/supposed to get a resolver? Or is it just a weird test thing?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634518604



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       It's just weird because it's different behavior than every other part of spark. `withColumn` and `withField` both append to the end. And currently before this PR and if the I applied the tweak `df1.unionByName(df2, True)` would not always have the same schema as `df2.unionByName(df1, True)`. It only sorts nested struct fields, not the top level attributes.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860056419






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r633682592



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       Can you clarify which step fails? I'm reading this test but can't think of how it can fail without this PR.

##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I see. From the JIRA, seems the problem is `b:struct<ba:string:bb:string>` becomes `b:struct<aa:string,bb:string>` due to some reasons. Can we fix the bug surgically without changing how the fields are ordered? Or do we have other reasons to change the behavior?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629777603



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -752,8 +786,50 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1: DataFrame = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2: DataFrame = spark.createDataFrame(

Review comment:
       Hah yep just copied that from somewhere else. I'm all for more type inferencing




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629773979



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       I just copied this from the `findNestedField` definition below in this class. I could also just remove the default, I think a test in the DataTypeSuite is the only thing not passing a value. Kept in case we didn't want to change the non-ResolveUnion places that uses this. Don't have a strong preference either way




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629621168



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {
     assert(col.dataType.isInstanceOf[StructType], "Only support StructType.")
 
     val resolver = conf.resolver
-    val missingFieldsOpt =
-      StructType.findMissingFields(col.dataType.asInstanceOf[StructType], target, resolver)
-
-    // We need to sort columns in result, because we might add another column in other side.
-    // E.g., we want to union two structs "a int, b long" and "a int, c string".
-    // If we don't sort, we will have "a int, b long, c string" and
-    // "a int, c string, b long", which are not compatible.
-    if (missingFieldsOpt.isEmpty) {
-      sortStructFields(col)
-    } else {
-      missingFieldsOpt.map { s =>
-        val struct = addFieldsInto(col, s.fields)
-        // Combines `WithFields`s to reduce expression tree.
-        val reducedStruct = struct.transformUp(OptimizeUpdateFields.optimizeUpdateFields)
-        val sorted = sortStructFieldsInWithFields(reducedStruct)
-        sorted
-      }.get
-    }
-  }
-
-  /**
-   * Adds missing fields recursively into given `col` expression. The missing fields are given
-   * in `fields`. For example, given `col` as "z struct<z:int, y:int>, x int", and `fields` is
-   * "z struct<w:long>, w string". This method will add a nested `z.w` field and a top-level
-   * `w` field to `col` and fill null values for them. Note that because we might also add missing
-   * fields at other side of Union, we must make sure corresponding attributes at two sides have
-   * same field order in structs, so when we adding missing fields, we will sort the fields based on
-   * field names. So the data type of returned expression will be
-   * "w string, x int, z struct<w:long, y:int, z:int>".
-   */
-  private def addFieldsInto(
-      col: Expression,
-      fields: Seq[StructField]): Expression = {
-    fields.foldLeft(col) { case (currCol, field) =>
-      field.dataType match {
-        case st: StructType =>
-          val resolver = conf.resolver
-          val colField = currCol.dataType.asInstanceOf[StructType]
-            .find(f => resolver(f.name, field.name))
-          if (colField.isEmpty) {
-            // The whole struct is missing. Add a null.
-            UpdateFields(currCol, field.name, Literal(null, st))
-          } else {
-            UpdateFields(currCol, field.name,
-              addFieldsInto(ExtractValue(currCol, Literal(field.name), resolver), st.fields))
-          }
-        case dt =>
-          UpdateFields(currCol, field.name, Literal(null, dt))
+    val colType = col.dataType.asInstanceOf[StructType]
+    val newStructFields = expectedFields.flatMap(expectedField => {

Review comment:
       style:
   
   ```scala
   expectedFields.flatMap { expectedField =>
      ...
   }
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840872115


   **[Test build #138528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138528/testReport)** for PR 32448 at commit [`b011c69`](https://github.com/apache/spark/commit/b011c696baaa8c84caf877961a37c2b0a61edca4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861930966


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139827/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-837635348


   > Should I remove the findMissingFields function that was added for the original method?
   
   Yea, please remove it as it is useless now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861829656


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861930966


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/139827/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840891914


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/43049/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman edited a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman edited a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860328810


   So @viirya's comment made me realize that `StructType.merge` isn't quite the right solution since it immediately fails on exact type mismatch and can't handle similar types like float/double. I updated things locally to not use it anymore and just resolve things based on name and let the types figure themselves out later like other unions.
   
   Separately, the `StructType.merge` not considering case sensitivity is still a bug that can crop up with unions (not even just unionByName with null filling).
   
   So I can either just push the minor update I have here to address the comment, or I can close this and open two separate PRs to address each individually.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861865124


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44355/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861827923


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44353/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629637776



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -752,8 +786,50 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1: DataFrame = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2: DataFrame = spark.createDataFrame(

Review comment:
       nit: df1, df2 don't need explicitly declared type `DataFrame`.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849367457






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629777957



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       The default resolver looks like for case-sensitive resolution. SparkSQL is case-insensitive by default. I'm afraid calling it by default parameter could lead to unexpected error. I'm not sure why `findNestedField` uses the default resolver. Removing default resolve is also good to me.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-836233745


   Can you also update https://github.com/apache/spark/blob/master/sql/core/src/main/scala/org/apache/spark/sql/Dataset.scala#L2078-L2082


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629616627



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -483,8 +483,8 @@ case class StructType(fields: Array[StructField]) extends DataType with Seq[Stru
    * 4. Otherwise, `this` and `that` are considered as conflicting schemas and an exception would be
    *    thrown.
    */
-  private[sql] def merge(that: StructType): StructType =
-    StructType.merge(this, that).asInstanceOf[StructType]
+  private[sql] def merge(that: StructType, resolver: Resolver = _ == _): StructType =

Review comment:
       Should we use `SQLConf.resolver` as default resolver?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629640357



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/types/StructType.scala
##########
@@ -555,32 +555,31 @@ object StructType extends AbstractDataType {
       case _ => dt
     }
 
-  private[sql] def merge(left: DataType, right: DataType): DataType =
+  private[sql] def merge(resolver: Resolver)(left: DataType, right: DataType): DataType =

Review comment:
       +1 for adding some tests for it.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-848476906


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138950/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849278564


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/43515/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861788921


   I pushed the small fix that moves the compatibility analysis after the union resolving, making ResolveUnion no longer use StructType.merge. I updated the title and description as well. Let me know if you want me to create separate PRs for each now instead


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861791977


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/44353/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629626137



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -181,7 +100,12 @@ object ResolveUnion extends Rule[LogicalPlan] {
             // like that. We will sort columns in the struct expression to make sure two sides of
             // union have consistent schema.
             aliased += foundAttr
-            Alias(addFields(foundAttr, target), foundAttr.name)()
+            val targetType = try {
+              target.merge(source, conf.resolver)
+            } catch { case e: Throwable =>
+              throw QueryExecutionErrors.failedToMergeIncompatibleSchemasError(target, source, e)

Review comment:
       Original exceptions looks more details than just `failedToMergeIncompatibleSchemasError`. I think it is better to let `merge` throw original exceptions.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849248625


   Rebased off latest master. I think everything has been addressed, let me know if there's any other thoughts or comments


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840980471


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/138528/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] cloud-fan commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
cloud-fan commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r634542944



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -743,17 +777,59 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       StructField("a", StringType)))
     val nestedStructValues2 = Row("b", "a")
 
-    val df1: DataFrame = spark.createDataFrame(
+    val df1 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType1))))
 
-    val df2: DataFrame = spark.createDataFrame(
+    val df2 = spark.createDataFrame(
       sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
       StructType(Seq(StructField("topLevelCol", nestedStructType2))))
 
     val union = df1.unionByName(df2, allowMissingColumns = true)
-    checkAnswer(union, Row(Row(null, "b")) :: Row(Row("a", "b")) :: Nil)
-    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRING, `b`: STRING>")
+    assert(union.schema.toDDL == "`topLevelCol` STRUCT<`b`: STRING, `a`: STRING>")
+    checkAnswer(union, Row(Row("b", null)) :: Row(Row("b", "a")) :: Nil)
+  }
+
+  test("SPARK-35290: Make unionByName null-filling behavior work with struct columns"
+      + " - sorting edge case") {
+    val nestedStructType1 = StructType(Seq(
+      StructField("b", StructType(Seq(
+        StructField("ba", StringType)
+      )))
+    ))
+    val nestedStructValues1 = Row(Row("ba"))
+
+    val nestedStructType2 = StructType(Seq(
+      StructField("a", StructType(Seq(
+        StructField("aa", StringType)
+      ))),
+      StructField("b", StructType(Seq(
+        StructField("bb", StringType)
+      )))
+    ))
+    val nestedStructValues2 = Row(Row("aa"), Row("bb"))
+
+    val df1 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues1) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType1))))
+
+    val df2 = spark.createDataFrame(
+      sparkContext.parallelize(Row(nestedStructValues2) :: Nil),
+      StructType(Seq(StructField("topLevelCol", nestedStructType2))))
+
+    var unionDf = df1.unionByName(df2, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` " +
+      "STRUCT<`b`: STRUCT<`ba`: STRING, `bb`: STRING>, `a`: STRUCT<`aa`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(Row("ba", null), null)) ::
+      Row(Row(Row(null, "bb"), Row("aa"))) :: Nil)
+
+    unionDf = df2.unionByName(df1, true)
+    assert(unionDf.schema.toDDL == "`topLevelCol` STRUCT<`a`: STRUCT<`aa`: STRING>, " +
+      "`b`: STRUCT<`bb`: STRING, `ba`: STRING>>")
+    checkAnswer(unionDf,
+      Row(Row(null, Row(null, "ba"))) ::
+      Row(Row(Row("aa"), Row("bb", null))) :: Nil)

Review comment:
       I see. So this PR mainly proposes to change the behavior, and fixing a bug is just a bonus.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-860128623


   retest this please


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629622363



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {
     assert(col.dataType.isInstanceOf[StructType], "Only support StructType.")
 
     val resolver = conf.resolver
-    val missingFieldsOpt =
-      StructType.findMissingFields(col.dataType.asInstanceOf[StructType], target, resolver)
-
-    // We need to sort columns in result, because we might add another column in other side.
-    // E.g., we want to union two structs "a int, b long" and "a int, c string".
-    // If we don't sort, we will have "a int, b long, c string" and
-    // "a int, c string, b long", which are not compatible.
-    if (missingFieldsOpt.isEmpty) {
-      sortStructFields(col)
-    } else {
-      missingFieldsOpt.map { s =>
-        val struct = addFieldsInto(col, s.fields)
-        // Combines `WithFields`s to reduce expression tree.
-        val reducedStruct = struct.transformUp(OptimizeUpdateFields.optimizeUpdateFields)
-        val sorted = sortStructFieldsInWithFields(reducedStruct)
-        sorted
-      }.get
-    }
-  }
-
-  /**
-   * Adds missing fields recursively into given `col` expression. The missing fields are given
-   * in `fields`. For example, given `col` as "z struct<z:int, y:int>, x int", and `fields` is
-   * "z struct<w:long>, w string". This method will add a nested `z.w` field and a top-level
-   * `w` field to `col` and fill null values for them. Note that because we might also add missing
-   * fields at other side of Union, we must make sure corresponding attributes at two sides have
-   * same field order in structs, so when we adding missing fields, we will sort the fields based on
-   * field names. So the data type of returned expression will be
-   * "w string, x int, z struct<w:long, y:int, z:int>".
-   */
-  private def addFieldsInto(
-      col: Expression,
-      fields: Seq[StructField]): Expression = {
-    fields.foldLeft(col) { case (currCol, field) =>
-      field.dataType match {
-        case st: StructType =>
-          val resolver = conf.resolver
-          val colField = currCol.dataType.asInstanceOf[StructType]
-            .find(f => resolver(f.name, field.name))
-          if (colField.isEmpty) {
-            // The whole struct is missing. Add a null.
-            UpdateFields(currCol, field.name, Literal(null, st))
-          } else {
-            UpdateFields(currCol, field.name,
-              addFieldsInto(ExtractValue(currCol, Literal(field.name), resolver), st.fields))
-          }
-        case dt =>
-          UpdateFields(currCol, field.name, Literal(null, dt))
+    val colType = col.dataType.asInstanceOf[StructType]
+    val newStructFields = expectedFields.flatMap(expectedField => {
+      val currentField = colType.fields.find(f => resolver(f.name, expectedField.name))
+
+      val newExpression = (currentField, expectedField.dataType) match {
+        case (Some(cf), expectedType: StructType) if cf.dataType.isInstanceOf[StructType] =>
+            val extractedValue = ExtractValue(col, Literal(cf.name), resolver)
+            val combinedStruct = addFields(extractedValue, expectedType.fields)
+            if (extractedValue.nullable) {
+              If(IsNull(extractedValue),
+                Literal(null, combinedStruct.dataType),
+                combinedStruct)
+            } else {
+              combinedStruct
+            }
+        case (Some(cf), _) =>
+          ExtractValue(col, Literal(cf.name), resolver)
+        case (_, expectedType) =>

Review comment:
       `None` instead of `_`. Explicitly matching looks better.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-834153063


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/42762/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r631516256



##########
File path: sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/ResolveUnion.scala
##########
@@ -21,136 +21,55 @@ import scala.collection.mutable
 
 import org.apache.spark.sql.AnalysisException
 import org.apache.spark.sql.catalyst.expressions._
-import org.apache.spark.sql.catalyst.optimizer.{CombineUnions, OptimizeUpdateFields}
+import org.apache.spark.sql.catalyst.optimizer.{CombineUnions}
 import org.apache.spark.sql.catalyst.plans.logical.{LogicalPlan, Project, Union}
 import org.apache.spark.sql.catalyst.rules.Rule
 import org.apache.spark.sql.catalyst.trees.AlwaysProcess
+import org.apache.spark.sql.errors._
 import org.apache.spark.sql.types._
 import org.apache.spark.sql.util.SchemaUtils
-import org.apache.spark.unsafe.types.UTF8String
 
 /**
  * Resolves different children of Union to a common set of columns.
  */
 object ResolveUnion extends Rule[LogicalPlan] {
-  /**
-   * This method sorts columns recursively in a struct expression based on column names.
-   */
-  private def sortStructFields(expr: Expression): Expression = {
-    val existingExprs = expr.dataType.asInstanceOf[StructType].fieldNames.zipWithIndex.map {
-      case (name, i) =>
-        val fieldExpr = GetStructField(KnownNotNull(expr), i)
-        if (fieldExpr.dataType.isInstanceOf[StructType]) {
-          (name, sortStructFields(fieldExpr))
-        } else {
-          (name, fieldExpr)
-        }
-    }.sortBy(_._1).flatMap(pair => Seq(Literal(pair._1), pair._2))
-
-    val newExpr = CreateNamedStruct(existingExprs)
-    if (expr.nullable) {
-      If(IsNull(expr), Literal(null, newExpr.dataType), newExpr)
-    } else {
-      newExpr
-    }
-  }
-
-  /**
-   * Assumes input expressions are field expression of `CreateNamedStruct`. This method
-   * sorts the expressions based on field names.
-   */
-  private def sortFieldExprs(fieldExprs: Seq[Expression]): Seq[Expression] = {
-    fieldExprs.grouped(2).map { e =>
-      Seq(e.head, e.last)
-    }.toSeq.sortBy { pair =>
-      assert(pair.head.isInstanceOf[Literal])
-      pair.head.eval().asInstanceOf[UTF8String].toString
-    }.flatten
-  }
-
-  /**
-   * This helper method sorts fields in a `UpdateFields` expression by field name.
-   */
-  private def sortStructFieldsInWithFields(expr: Expression): Expression = expr transformUp {
-    case u: UpdateFields if u.resolved =>
-      u.evalExpr match {
-        case i @ If(IsNull(_), _, CreateNamedStruct(fieldExprs)) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          i.copy(trueValue = Literal(null, newStruct.dataType), falseValue = newStruct)
-        case CreateNamedStruct(fieldExprs) =>
-          val sorted = sortFieldExprs(fieldExprs)
-          val newStruct = CreateNamedStruct(sorted)
-          newStruct
-        case other =>
-          throw new IllegalStateException(s"`UpdateFields` has incorrect expression: $other. " +
-            "Please file a bug report with this error message, stack trace, and the query.")
-      }
-  }
-
   /**
    * Adds missing fields recursively into given `col` expression, based on the target `StructType`.
    * This is called by `compareAndAddFields` when we find two struct columns with same name but
    * different nested fields. This method will find out the missing nested fields from `col` to
    * `target` struct and add these missing nested fields. Currently we don't support finding out
    * missing nested fields of struct nested in array or struct nested in map.
    */
-  private def addFields(col: NamedExpression, target: StructType): Expression = {
+  private def addFields(col: Expression, expectedFields: Seq[StructField]): Expression = {

Review comment:
       Done, let me know how it sounds




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] Kimahriman commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
Kimahriman commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r632140770



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")
+      checkAnswer(unionDf,
+        Row(0, Row(0, 1, Row(1, "2", null, null))) ::
+          Row(1, Row(1, 2, Row(null, null, 2, 3L))) :: Nil)
 
       unionDf = df2.unionByName(df1, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L, null))) ::
-          Row(0, Row(0, 1, Row(null, 1, null, "2"))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT, `c`: STRING>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null, null))) ::
+          Row(0, Row(0, 1, Row(null, null, 1, "2"))) :: Nil)
 
       val df3 = Seq((2, UnionClass1b(2, 3L, UnionClass3(4, 5L)))).toDF("id", "a")
       unionDf = df2.unionByName(df3, true)
-      checkAnswer(unionDf,
-        Row(1, Row(1, 2, Row(2, null, 3L))) ::
-          Row(2, Row(2, 3, Row(null, 4, 5L))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT>>")
+          "`nested`: STRUCT<`A`: INT, `b`: BIGINT, `a`: INT>>")
+      checkAnswer(unionDf,
+        Row(1, Row(1, 2, Row(2, 3L, null))) ::
+          Row(2, Row(2, 3, Row(null, 5L, 4))) :: Nil)
     }
   }
 
+  test("SPARK-32376: Make unionByName null-filling behavior work with struct columns" +
+      " - case-insensitive cases") {

Review comment:
       I think I just created this test to check cases where the struct column itself is in different casing, but I don't think it failed before this PR (other than ordering changes as a result of this PR). The one that does fail before this PR is the one tagged 35290. I can either update the tag on this one, or update the one on line 676 to also test the different casing on the struct column. I think I just copy pasted and left the tag as is




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840324232


   **[Test build #138481 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138481/testReport)** for PR 32448 at commit [`93b47d3`](https://github.com/apache/spark/commit/93b47d3f190369afdf5a2a5ae0ec0c6054b56c1b).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-840872115


   **[Test build #138528 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/138528/testReport)** for PR 32448 at commit [`b011c69`](https://github.com/apache/spark/commit/b011c696baaa8c84caf877961a37c2b0a61edca4).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629072673



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       Can we update migration guide (https://github.com/apache/spark/blob/master/docs/sql-migration-guide.md)?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #32448: [SPARK-35290][SQL] Append new nested struct fields rather than sort for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-861827923


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/44353/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-849344876


   **[Test build #139000 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/139000/testReport)** for PR 32448 at commit [`ba99753`](https://github.com/apache/spark/commit/ba997539146f4b0bb53d4b62e6b097361db35e77).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] HyukjinKwon commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-833186519


   cc @viirya FYI


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on a change in pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #32448:
URL: https://github.com/apache/spark/pull/32448#discussion_r629632334



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/DataFrameSetOperationsSuite.scala
##########
@@ -707,32 +710,63 @@ class DataFrameSetOperationsSuite extends QueryTest with SharedSparkSession {
       val df2 = Seq((1, UnionClass1c(1, 2L, UnionClass4(2, 3L)))).toDF("id", "a")
 
       var unionDf = df1.unionByName(df2, true)
-      checkAnswer(unionDf,
-        Row(0, Row(0, 1, Row(null, 1, null, "2"))) ::
-          Row(1, Row(1, 2, Row(2, null, 3L, null))) :: Nil)
       assert(unionDf.schema.toDDL ==
         "`id` INT,`a` STRUCT<`a`: INT, `b`: BIGINT, " +
-          "`nested`: STRUCT<`A`: INT, `a`: INT, `b`: BIGINT, `c`: STRING>>")
+          "`nested`: STRUCT<`a`: INT, `c`: STRING, `A`: INT, `b`: BIGINT>>")

Review comment:
       yes




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] viirya commented on pull request #32448: [SPARK-35290][SQL] Use StructType merging for unionByName with null filling

Posted by GitBox <gi...@apache.org>.
viirya commented on pull request #32448:
URL: https://github.com/apache/spark/pull/32448#issuecomment-843859830


   > The code change itself looks good. It's better to clarify why there was a failure before. Sorting the nested fields shouldn't lead to a bug that `b:struct<ba:string:bb:string>` becomes `b:struct<aa:string,bb:string>`.
   
   I'm not sure about it. @Kimahriman Do you have idea? If not, I may take some time debugging it.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org