You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/01/25 22:30:07 UTC

[GitHub] [spark] xkrogen opened a new pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

xkrogen opened a new pull request #31332:
URL: https://github.com/apache/spark/pull/31332


   ### What changes were proposed in this pull request?
   Change `AvroSuite."Ignore corrupt Avro file if flag IGNORE_CORRUPT_FILES"` to use `episodesAvro`, which is loaded as a resource using the classloader, instead of trying to read `episodes.avro` directly from a relative file path.
   
   ### Why are the changes needed?
   This is the proper way to read resource files, and currently this test will fail when called from my IntelliJ IDE, though it will succeed when called from Maven/sbt, presumably due to different working directory handling.
   
   ### Does this PR introduce _any_ user-facing change?
   No, unit test only.
   
   ### How was this patch tested?
   Previous failure from IntelliJ:
   ```
   Source 'src/test/resources/episodes.avro' does not exist
   java.io.FileNotFoundException: Source 'src/test/resources/episodes.avro' does not exist
   	at org.apache.commons.io.FileUtils.checkFileRequirements(FileUtils.java:1405)
   	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1072)
   	at org.apache.commons.io.FileUtils.copyFile(FileUtils.java:1040)
   	at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34(AvroSuite.scala:397)
   	at org.apache.spark.sql.avro.AvroSuite.$anonfun$new$34$adapted(AvroSuite.scala:388)
   ```
   Now it succeeds.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767173384


   Kubernetes integration test starting
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on pull request #31332: [SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-768394879


   Good catch on the PR title as well, thank you for fixing 🙏🏼 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31332: [SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-768088526


   Merged to master, branch-3.1, branch-3.0 and branch-2.4.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767157781


   **[Test build #134466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134466/testReport)** for PR 31332 at commit [`4401517`](https://github.com/apache/spark/commit/4401517d551ceea5456d8a9cccdaba725b9504c2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on pull request #31332: [SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-768394279


   Thanks @HyukjinKwon !


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #31332: [SPARK-34231][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31332:
URL: https://github.com/apache/spark/pull/31332


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767157781


   **[Test build #134466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134466/testReport)** for PR 31332 at commit [`4401517`](https://github.com/apache/spark/commit/4401517d551ceea5456d8a9cccdaba725b9504c2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767177039






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767177039






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] xkrogen commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
xkrogen commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767712920


   @mridulm would you be able to help review this small change?


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767157781


   **[Test build #134466 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134466/testReport)** for PR 31332 at commit [`4401517`](https://github.com/apache/spark/commit/4401517d551ceea5456d8a9cccdaba725b9504c2).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767177039


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134466/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767157781






----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767198375


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39052/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767198375


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/39052/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767169227


   **[Test build #134466 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/134466/testReport)** for PR 31332 at commit [`4401517`](https://github.com/apache/spark/commit/4401517d551ceea5456d8a9cccdaba725b9504c2).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767184061


   Kubernetes integration test status success
   URL: https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder-K8s/39052/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31332: [SPARK-34241][AVRO][TEST] Make proper use of resource file within AvroSuite test case

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31332:
URL: https://github.com/apache/spark/pull/31332#issuecomment-767177039


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/134466/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org