You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/07/12 22:51:15 UTC

[GitHub] [spark] xkrogen opened a new pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

xkrogen opened a new pull request #33308:
URL: https://github.com/apache/spark/pull/33308


   ### What changes were proposed in this pull request?
   This unifies struct schema mismatch-handling logic between `AvroSerializer` and `AvroDeserializer`, pushing it into `AvroUtils` which is used by both. The newly unified exception-handling logic is updated to provide more contextual information in error messages. When a schema mismatch is found, previously we would only report the first missing field that is found, but there may be any others as well, which can make it less clear what exactly is going wrong. Now, we will report on all missing fields.
   
   ### Why are the changes needed?
   While working on #31490, we discussed that there is room for improvement in how schema mismatch errors are reported ([comment1](https://github.com/apache/spark/pull/31490#discussion_r659970793), [comment2](https://github.com/apache/spark/pull/31490#issuecomment-869866848)). Additionally, the logic between `AvroSerializer` and `AvroDeserializer` was quite similar for handling these issues, but didn't share common code, causing duplication and making it harder to see exactly what differences existed between the two.
   
   ### Does this PR introduce _any_ user-facing change?
   Some error messages when matching Catalyst struct schemas against Avro record schemas now include more information.
   
   ### How was this patch tested?
   New unit tests added.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r676774212



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' (at position 0) in Avro schema at field 'foo'",

Review comment:
       Yes I think you're right. I think the issue is that I was trying too hard to unify the code paths / exception messages for the by-name and by-position cases. I've updated this and I think the messages should be more readable now.
   
   In the case of positional matching, I still report the field names of both the SQL and Avro fields, since they may be different. I can add logic to omit the second instance of the name if they're matching, but I don't think it's necessary. Lmk your thoughts.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672363796



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       The result of this method is complicated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886922981


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886961590


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46169/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885069213


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141506/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885038335


   **[Test build #141506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141506/testReport)** for PR 33308 at commit [`5904fea`](https://github.com/apache/spark/commit/5904feaa7ad16e4dcadc9d04b713609fc2ab9dc1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882832486


   Thanks for the suggestions @gengliangwang ! I've refactored a bit and taken your suggestions. I agree that the code previously was confusing, I hope it should be more clear now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884541195


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45959/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882631956


   @xkrogen I am sorry for the late reply. I think my major concern is the code becomes more complicated. I think it can be better by either:
   * Simply adding new functions which go over the schema and validate the schema without storing `fieldPairs`. 
   Or
   * Create `AvroMatchedField` to make the function getMatchedFields readable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672363796



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       The result of this method is complicated.

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       TBH, this line is a bit hard to understand...

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Let's create a new case class for this:
   ```
   case class AvroMatchedField(avroField Schema.Field, catalystField StructField, catalystPosition int)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887805490


   Kubernetes integration test status failure
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46238/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887773257


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46238/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878673880


   **[Test build #140942 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140942/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141260/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882551301


   Jenkins, retest this please.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141260/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-890995071


   Thanks, merging to master


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884462301


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45952/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878681383


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45454/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882700920


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45774/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672365360



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       TBH, this line is a bit hard to understand...




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884495194


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141440/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886926285


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46169/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674723801



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +242,40 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    /** Get the fields which have matching equivalents in both Avro and Catalyst schemas. */
+    def getMatchedFields: Seq[AvroMatchedField] = catalystSchema.zipWithIndex.flatMap {

Review comment:
       let's store this in a `val`. My previous comment for "not storing" is for adding new code for validation only.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882885001






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882551301






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945


   **[Test build #141260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141260/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884462070


   **[Test build #141440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141440/testReport)** for PR 33308 at commit [`fe27230`](https://github.com/apache/spark/commit/fe27230de77747d5290a66d1dc03e7413e6cf47d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885080296


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46023/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886884899


   **[Test build #141653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141653/testReport)** for PR 33308 at commit [`44fe81b`](https://github.com/apache/spark/commit/44fe81bcee8571ba9668843a6075bf0223c754bb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672589300



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Great suggestion, thanks!




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-891323922


   Thanks a lot for all of the great comments @gengliangwang! I'm very happy with the final version.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884491149


   **[Test build #141440 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141440/testReport)** for PR 33308 at commit [`fe27230`](https://github.com/apache/spark/commit/fe27230de77747d5290a66d1dc03e7413e6cf47d).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r676014641



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' (at position 0) in Avro schema at field 'foo'",
       nonnullCatalyst)
     assertFailedConversionMessage(avro, Deserializer, BY_POSITION,
-      "Cannot find non-nullable field at position 1 (field 'foo.baz') in Avro schema.",
+      "Cannot find field at position 1 ('foo.baz') in Avro schema at field 'foo'",

Review comment:
       ok I see. I didn't notice that the original error message has such a wording order too.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886884899


   **[Test build #141653 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141653/testReport)** for PR 33308 at commit [`44fe81b`](https://github.com/apache/spark/commit/44fe81bcee8571ba9668843a6075bf0223c754bb).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang edited a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882631956


   @xkrogen I am sorry for the late reply. I think my major concern is the code becomes more complicated. I think it can be better by either:
   * Simply adding new functions which go over the schema and validate the schema without storing `fieldPairs`. 
   
   Or
   * Create `AvroMatchedField` to make the function getMatchedFields readable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882832486


   Thanks for the suggestions @gengliangwang ! I've refactored a bit and taken your suggestions. I agree that the code previously was confusing, I hope it should be more clear now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882852154


   **[Test build #141279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141279/testReport)** for PR 33308 at commit [`dde54a3`](https://github.com/apache/spark/commit/dde54a3ab1f35f60db01e2e73cf598c032369856).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674208555



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -1403,7 +1403,8 @@ abstract class AvroSuite
           df.write.option("avroSchema", avroSchema).format("avro").save(s"$tempDir/save2")
         }
         assertExceptionMsg[IncompatibleSchemaException](e,
-          "Cannot find field 'FOO' (at position 0) in Avro schema at top-level record")
+          "Cannot find SQL field(s) in Avro top-level record: " +

Review comment:
       Ok, makes sense. I also agree that trying to phrase it properly when reporting multiple errors was more challenging. I think it's a tradeoff between more information vs. understandability of that information, and I'm find with going for better understandability. Updated.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882652447


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45774/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885038335


   **[Test build #141506 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141506/testReport)** for PR 33308 at commit [`5904fea`](https://github.com/apache/spark/commit/5904feaa7ad16e4dcadc9d04b713609fc2ab9dc1).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878684216






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886961590


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46169/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886922981


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141653/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882551301






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672589300



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Great suggestion, thanks!

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       Agreed, I refactored this to make it more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674952903



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' (at position 0) in Avro schema at field 'foo'",
       nonnullCatalyst)
     assertFailedConversionMessage(avro, Deserializer, BY_POSITION,
-      "Cannot find non-nullable field at position 1 (field 'foo.baz') in Avro schema.",
+      "Cannot find field at position 1 ('foo.baz') in Avro schema at field 'foo'",

Review comment:
       Sorry I don't understand, are you saying you think the expected message is incorrect? One thing to note is that this test is on `extraNonnullCatalyst` as the Catalyst schema, whereas the previous test is on `nonnullCatalyst`. We need an extra field because `CATALYST_STRUCT` and `nonnullCatalyst` have the same field count as `avro`, so when doing `BY_POSITION` it will succeed. Thus we use `extraNonnullCatalyst` which has an extra field `foo.baz` at position 1.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang edited a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882631956


   @xkrogen I am sorry for the late reply. I think my major concern is the code becomes more complicated. I think it can be better by either:
   * Simply adding new functions which go over the schema and validate the schema without storing `fieldPairs`. 
   
   Or
   * Create `AvroMatchedField` to make the function getMatchedFields readable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884571764


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45959/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884462070


   **[Test build #141440 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141440/testReport)** for PR 33308 at commit [`fe27230`](https://github.com/apache/spark/commit/fe27230de77747d5290a66d1dc03e7413e6cf47d).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674954527



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +242,40 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    /** Get the fields which have matching equivalents in both Avro and Catalyst schemas. */
+    def getMatchedFields: Seq[AvroMatchedField] = catalystSchema.zipWithIndex.flatMap {

Review comment:
       Sure makes sense.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887718418


   **[Test build #141725 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141725/testReport)** for PR 33308 at commit [`e7e7f17`](https://github.com/apache/spark/commit/e7e7f174aaef0dd2bef7a7a0cc106ff01ac170ef).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885037105


   Pushed up a new commit addressing @gengliangwang's comment to store the matched fields as a `val`.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674723012



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' (at position 0) in Avro schema at field 'foo'",
       nonnullCatalyst)
     assertFailedConversionMessage(avro, Deserializer, BY_POSITION,
-      "Cannot find non-nullable field at position 1 (field 'foo.baz') in Avro schema.",
+      "Cannot find field at position 1 ('foo.baz') in Avro schema at field 'foo'",

Review comment:
       at position 1 ('foo.baz') 
   =>
   'foo.bar' (at position 0)




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878664649


   **[Test build #140942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140942/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885061558


   **[Test build #141506 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141506/testReport)** for PR 33308 at commit [`5904fea`](https://github.com/apache/spark/commit/5904feaa7ad16e4dcadc9d04b713609fc2ab9dc1).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672386792



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Let's create a new case class for this:
   ```
   case class AvroMatchedField(avroField Schema.Field, catalystField StructField, catalystPosition int)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r677112782



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' in Avro schema",
       nonnullCatalyst)
     assertFailedConversionMessage(avro, Deserializer, BY_POSITION,
-      "Cannot find non-nullable field at position 1 (field 'foo.baz') in Avro schema.",
+      "Cannot find field at position 1 of field 'foo' from Avro schema",

Review comment:
       Shall we mention by-position match?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887809117


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46238/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887693487


   **[Test build #141725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141725/testReport)** for PR 33308 at commit [`e7e7f17`](https://github.com/apache/spark/commit/e7e7f174aaef0dd2bef7a7a0cc106ff01ac170ef).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang edited a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang edited a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882631956


   @xkrogen I am sorry for the late reply. I think my major concern is the code becomes more complicated. I think it can be better by either:
   * Simply adding new functions which go over the schema and validate the schema without storing `fieldPairs`. 
   
   Or
   * Create `AvroMatchedField` to make the function getMatchedFields readable.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882873578


   **[Test build #141279 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141279/testReport)** for PR 33308 at commit [`dde54a3`](https://github.com/apache/spark/commit/dde54a3ab1f35f60db01e2e73cf598c032369856).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds the following public classes _(experimental)_:
     * `  case class AvroMatchedField(`


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886946947


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/46169/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945


   **[Test build #141260 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141260/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882589915


   **[Test build #141260 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141260/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r677642548



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' in Avro schema",

Review comment:
       Since by-name is the default, I'm thinking to omit that, and only mention if it's using positional matching. WDYT?

##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' in Avro schema",
       nonnullCatalyst)
     assertFailedConversionMessage(avro, Deserializer, BY_POSITION,
-      "Cannot find non-nullable field at position 1 (field 'foo.baz') in Avro schema.",
+      "Cannot find field at position 1 of field 'foo' from Avro schema",

Review comment:
       good idea




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672589300



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Great suggestion, thanks!

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       Agreed, I refactored this to make it more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-885069213


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141506/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884559124


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45959/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884462325


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45952/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887693487


   **[Test build #141725 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141725/testReport)** for PR 33308 at commit [`e7e7f17`](https://github.com/apache/spark/commit/e7e7f174aaef0dd2bef7a7a0cc106ff01ac170ef).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884415829


   **[Test build #141433 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141433/testReport)** for PR 33308 at commit [`8821975`](https://github.com/apache/spark/commit/8821975b34ca25d56af23a13f688f88ece63d9ab).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang closed pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang closed pull request #33308:
URL: https://github.com/apache/spark/pull/33308


   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887731432


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141725/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884495194


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141440/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884571764


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45959/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882832486


   Thanks for the suggestions @gengliangwang ! I've refactored a bit and taken your suggestions. I agree that the code previously was confusing, I hope it should be more clear now.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r674113916



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -1403,7 +1403,8 @@ abstract class AvroSuite
           df.write.option("avroSchema", avroSchema).format("avro").save(s"$tempDir/save2")
         }
         assertExceptionMsg[IncompatibleSchemaException](e,
-          "Cannot find field 'FOO' (at position 0) in Avro schema at top-level record")
+          "Cannot find SQL field(s) in Avro top-level record: " +

Review comment:
       The original error message is clear about the Avro schema doesn't have the field, while the new one makes me confused about which schema has missed field, Avro or Catalyst.
   
   Also, showing one error at a time is OK.
   
   @xkrogen Shall we keep the error message as it is? This PR can be just moving the schema validation to the `AvroSchemaHelper`. I am sorry but it's my personal taste as a non-native speaker. 




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882611375






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884300525


   Overall LGTM. Thanks for the work


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r676014909



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' (at position 0) in Avro schema at field 'foo'",

Review comment:
       The wording `at field 'foo'` seems unnecessary, as we already show the full field name 'foo.bar'.
   Also, "Cannot find **field** .. in Avro schema at **field** .." sounds odd.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882881666


   Kubernetes integration test unable to build dist.
   
   exiting with code: 1
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45793/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884420126


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141433/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672363796



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       The result of this method is complicated.

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       TBH, this line is a bit hard to understand...

##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {

Review comment:
       Let's create a new case class for this:
   ```
   case class AvroMatchedField(avroField Schema.Field, catalystField StructField, catalystPosition int)
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882679432


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/45774/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884462325


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45952/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882885001






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882565945






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884379037


   Thanks for the insightful feedback @gengliangwang ! Updated with more simplification and less change to the messages.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] gengliangwang commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
gengliangwang commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r677112492



##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSerdeSuite.scala
##########
@@ -86,18 +86,18 @@ class AvroSerdeSuite extends SparkFunSuite {
     // deserialize should have no issues when 'bar' is nullable but fail when it is nonnull
     Deserializer.create(CATALYST_STRUCT, avro, BY_NAME)
     assertFailedConversionMessage(avro, Deserializer, BY_NAME,
-      "Cannot find non-nullable field 'foo.bar' (at position 0) in Avro schema.",
+      "Cannot find field 'foo.bar' in Avro schema",

Review comment:
       Shall we mention "by-name" match here?
   ```
   Cannot find field 'foo.bar' in Avro schema with by-name match
   ```




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887809117


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/46238/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882852154


   **[Test build #141279 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141279/testReport)** for PR 33308 at commit [`dde54a3`](https://github.com/apache/spark/commit/dde54a3ab1f35f60db01e2e73cf598c032369856).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878684216






-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884382803


   **[Test build #141433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141433/testReport)** for PR 33308 at commit [`8821975`](https://github.com/apache/spark/commit/8821975b34ca25d56af23a13f688f88ece63d9ab).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878648554


   cc @gengliangwang @mridulm 


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-878664649


   **[Test build #140942 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/140942/testReport)** for PR 33308 at commit [`8a4f005`](https://github.com/apache/spark/commit/8a4f0053978d34e5b20d18ddc61dbda8605cbdf2).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] xkrogen commented on a change in pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
xkrogen commented on a change in pull request #33308:
URL: https://github.com/apache/spark/pull/33308#discussion_r672601738



##########
File path: external/avro/src/main/scala/org/apache/spark/sql/avro/AvroUtils.scala
##########
@@ -228,6 +236,38 @@ private[sql] object AvroUtils extends Logging {
       .groupBy(_.name.toLowerCase(Locale.ROOT))
       .mapValues(_.toSeq) // toSeq needed for scala 2.13
 
+    private[this] val fieldPairs = catalystSchema
+      .zipWithIndex
+      .map(fieldWithPos => (fieldWithPos, getAvroField(fieldWithPos._1.name, fieldWithPos._2)))
+
+    /**
+     * Get the fields which have matching equivalents in both Avro and Catalyst schemas.
+     *
+     * @return A sequence of `(catalystField, catalystPosition, avroField)` tuples.
+     */
+    def getMatchedFields: Seq[(StructField, Int, Schema.Field)] = fieldPairs.flatMap {
+      case ((sqlField, sqlPos), Some(avroField)) => Some((sqlField, sqlPos, avroField))
+      case (_, None) => None
+    }
+
+    def assertNoExtraSqlFields(includeNullable: Boolean): Unit =
+      fieldPairs.filter(_._2.isEmpty).map(_._1).filter(includeNullable || !_._1.nullable) match {

Review comment:
       Agreed, I refactored this to make it more clear.




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884382803


   **[Test build #141433 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141433/testReport)** for PR 33308 at commit [`8821975`](https://github.com/apache/spark/commit/8821975b34ca25d56af23a13f688f88ece63d9ab).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-882700920


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/45774/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins removed a comment on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-887731432


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141725/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] AmplabJenkins commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-884420126


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/141433/
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org


[GitHub] [spark] SparkQA commented on pull request #33308: [SPARK-35918][AVRO] Unify schema mismatch handling for read/write and enhance error messages

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #33308:
URL: https://github.com/apache/spark/pull/33308#issuecomment-886907154


   **[Test build #141653 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/141653/testReport)** for PR 33308 at commit [`44fe81b`](https://github.com/apache/spark/commit/44fe81bcee8571ba9668843a6075bf0223c754bb).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org