You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/10/22 05:56:32 UTC
[GitHub] [spark] LuciferYang opened a new pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
LuciferYang opened a new pull request #34364:
URL: https://github.com/apache/spark/pull/34364
### What changes were proposed in this pull request?
The `external/avro` module test with Java 17 has 2 UT failed, the reason for the failure is that the content of the error message used for assertion has changed from
```
Caused by: java.lang.NullPointerException: null of string in string in field Name of test_schema in test_schema
```
to
```
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
```
This change should be caused by `JEP 358: Helpful NullPointerExceptions`.
So the change of this pr is splits one assertion into two assertions to be compatible with different Java versions.
### Why are the changes needed?
Pass UT with JDK 17
### Does this PR introduce _any_ user-facing change?
No
### How was this patch tested?
- Pass the Jenkins or GitHub Action
- Manual test use Java 17
`mvn clean install -pl external/avro`
**Before**
```
Run completed in 49 seconds, 212 milliseconds.
Total number of tests run: 283
Suites: completed 14, aborted 0
Tests: succeeded 281, failed 2, canceled 0, ignored 2, pending 0
*** 2 TESTS FAILED ***
```
**After**
```
Run completed in 42 seconds, 825 milliseconds.
Total number of tests run: 283
Suites: completed 14, aborted 0
Tests: succeeded 283, failed 0, canceled 0, ignored 2, pending 0
All tests passed.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34364:
URL: https://github.com/apache/spark/pull/34364#discussion_r734267797
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -1071,7 +1071,8 @@ abstract class AvroSuite
.write.format("avro").option("avroSchema", avroSchema)
.save(s"$tempDir/${UUID.randomUUID()}")
}.getCause.getMessage
- assert(message.contains("Caused by: java.lang.NullPointerException: " +
+ assert(message.contains("Caused by: java.lang.NullPointerException: "))
+ assert(message.contains(
Review comment:
The whole message as follows:
```
"Job aborted due to stage failure: Task 1 in stage 144.0 failed 1 times, most recent failure: Lost task 1.0 in stage 144.0 (TID 250) (localhost executor driver): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:516)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:345)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$20(FileFormatWriter.scala:252)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:317)
at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:84)
at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:62)
at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:84)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:328)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1502)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:335)
... 9 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:184)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:160)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
... 18 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null
at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:68)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221)
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210)
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
... 20 more
```
so we can't assert with `startWith` and `endWith`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949323115
**[Test build #144530 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144530/testReport)** for PR 34364 at commit [`a0c7118`](https://github.com/apache/spark/commit/a0c7118cf3a48f0f465b1f0cd949a300e7345282).
* This patch passes all tests.
* This patch merges cleanly.
* This patch adds no public classes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949334037
Kubernetes integration test starting
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49001/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA removed a comment on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949309560
**[Test build #144530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144530/testReport)** for PR 34364 at commit [`a0c7118`](https://github.com/apache/spark/commit/a0c7118cf3a48f0f465b1f0cd949a300e7345282).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949379409
Merged to master.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949374600
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49001/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949309560
**[Test build #144530 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/144530/testReport)** for PR 34364 at commit [`a0c7118`](https://github.com/apache/spark/commit/a0c7118cf3a48f0f465b1f0cd949a300e7345282).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949333658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949333658
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/144530/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] SparkQA commented on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949374555
Kubernetes integration test status failure
URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/49001/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] AmplabJenkins removed a comment on pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #34364:
URL: https://github.com/apache/spark/pull/34364#issuecomment-949374600
Refer to this link for build results (access rights to CI server needed):
https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/49001/
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] HyukjinKwon closed pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #34364:
URL: https://github.com/apache/spark/pull/34364
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] LuciferYang commented on a change in pull request #34364: [SPARK-37073][AVRO][TEST] Pass all UTs in `external/avro` with Java 17
Posted by GitBox <gi...@apache.org>.
LuciferYang commented on a change in pull request #34364:
URL: https://github.com/apache/spark/pull/34364#discussion_r734267797
##########
File path: external/avro/src/test/scala/org/apache/spark/sql/avro/AvroSuite.scala
##########
@@ -1071,7 +1071,8 @@ abstract class AvroSuite
.write.format("avro").option("avroSchema", avroSchema)
.save(s"$tempDir/${UUID.randomUUID()}")
}.getCause.getMessage
- assert(message.contains("Caused by: java.lang.NullPointerException: " +
+ assert(message.contains("Caused by: java.lang.NullPointerException: "))
+ assert(message.contains(
Review comment:
The whole error message as follows:
```
"Job aborted due to stage failure: Task 1 in stage 144.0 failed 1 times, most recent failure: Lost task 1.0 in stage 144.0 (TID 250) (localhost executor driver): org.apache.spark.SparkException: Task failed while writing rows.
at org.apache.spark.sql.errors.QueryExecutionErrors$.taskFailedWhileWritingRowsError(QueryExecutionErrors.scala:516)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:345)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$write$20(FileFormatWriter.scala:252)
at org.apache.spark.scheduler.ResultTask.runTask(ResultTask.scala:90)
at org.apache.spark.scheduler.Task.run(Task.scala:136)
at org.apache.spark.executor.Executor$TaskRunner.$anonfun$run$3(Executor.scala:507)
at org.apache.spark.util.Utils$.tryWithSafeFinally(Utils.scala:1468)
at org.apache.spark.executor.Executor$TaskRunner.run(Executor.scala:510)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:833)
Caused by: org.apache.avro.file.DataFileWriter$AppendWriteException: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:317)
at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:84)
at org.apache.spark.sql.avro.SparkAvroKeyRecordWriter.write(SparkAvroKeyOutputFormat.java:62)
at org.apache.spark.sql.avro.AvroOutputWriter.write(AvroOutputWriter.scala:84)
at org.apache.spark.sql.execution.datasources.SingleDirectoryDataWriter.write(FileFormatDataWriter.scala:175)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithMetrics(FileFormatDataWriter.scala:85)
at org.apache.spark.sql.execution.datasources.FileFormatDataWriter.writeWithIterator(FileFormatDataWriter.scala:92)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.$anonfun$executeTask$1(FileFormatWriter.scala:328)
at org.apache.spark.util.Utils$.tryWithSafeFinallyAndFailureCallbacks(Utils.scala:1502)
at org.apache.spark.sql.execution.datasources.FileFormatWriter$.executeTask(FileFormatWriter.scala:335)
... 9 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null of string in string in field Name of test_schema in test_schema
at org.apache.avro.generic.GenericDatumWriter.npe(GenericDatumWriter.java:184)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:160)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:73)
at org.apache.avro.file.DataFileWriter.append(DataFileWriter.java:314)
... 18 more
Caused by: java.lang.NullPointerException: Cannot invoke "Object.getClass()" because "datum" is null
at org.apache.avro.specific.SpecificDatumWriter.writeString(SpecificDatumWriter.java:68)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:151)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
at org.apache.avro.generic.GenericDatumWriter.writeField(GenericDatumWriter.java:221)
at org.apache.avro.specific.SpecificDatumWriter.writeField(SpecificDatumWriter.java:101)
at org.apache.avro.reflect.ReflectDatumWriter.writeField(ReflectDatumWriter.java:184)
at org.apache.avro.generic.GenericDatumWriter.writeRecord(GenericDatumWriter.java:210)
at org.apache.avro.specific.SpecificDatumWriter.writeRecord(SpecificDatumWriter.java:83)
at org.apache.avro.generic.GenericDatumWriter.writeWithoutConversion(GenericDatumWriter.java:131)
at org.apache.avro.generic.GenericDatumWriter.write(GenericDatumWriter.java:83)
at org.apache.avro.reflect.ReflectDatumWriter.write(ReflectDatumWriter.java:158)
... 20 more
```
so we can't assert with `startWith` and `endWith`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org