You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/03/02 05:38:28 UTC

[GitHub] [spark] dongjoon-hyun opened a new pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

dongjoon-hyun opened a new pull request #31697:
URL: https://github.com/apache/spark/pull/31697


   ### What changes were proposed in this pull request?
   
   1. This PR aims to ignore ORC encryption tests when ORC shim is loaded by old Hadoop library by some other tests. The test coverage is preserved by Jenkins SBT runs and GitHub Action jobs. This PR only aims to recover Maven Jenkins jobs.
   2. Also
   
   ### Why are the changes needed?
   
   Currently, Maven test fails when it runs in a batch mode because `HadoopShimsPre2_3$NullKeyProvider` is loaded.
   
   **COMMAND**
   ```
   $ mvn test -pl sql/core --am -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.datasources.orc.OrcV1QuerySuite,org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite
   ```
   
   **BEFORE**
   ```
   - Write and read an encrypted table *** FAILED ***
   ...
     Cause: org.apache.spark.SparkException: Job aborted due to stage failure: Task 0 in stage 1.0 failed 1 times, most recent failure: Lost task 0.0 in stage 1.0 (TID 1) (localhost executor driver): java.lang.IllegalArgumentException: Unknown key pii
   	at org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider.getCurrentKeyVersion(HadoopShimsPre2_3.java:71)
   	at org.apache.orc.impl.WriterImpl.getKey(WriterImpl.java:871)
   ```
   
   **AFTER**
   ```
   OrcV1QuerySuite
   ...
   OrcEncryptionSuite:
   - Write and read an encrypted file !!! CANCELED !!!
     [] was empty org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider@1b705f65 doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:39)
   - Write and read an encrypted table !!! CANCELED !!!
     [] was empty org.apache.orc.impl.HadoopShimsPre2_3$NullKeyProvider@22adeee1 doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:67)
   ```
   
   ### Does this PR introduce _any_ user-facing change?
   
   No.
   
   ### How was this patch tested?
   
   Pass the Jenkins Maven tests.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788642908


   **[Test build #135625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135625/testReport)** for PR 31697 at commit [`27c87a5`](https://github.com/apache/spark/commit/27c87a56c55217e992669634577559bab246526f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] maropu commented on a change in pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
maropu commented on a change in pull request #31697:
URL: https://github.com/apache/spark/pull/31697#discussion_r585293236



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcEncryptionSuite.scala
##########
@@ -30,6 +32,11 @@ class OrcEncryptionSuite extends OrcTest with SharedSparkSession {
     Row(null, "841626795E7D351555B835A002E3BF10669DE9B81C95A3D59E10865AC37EA7C3", "Dongjoon Hyun")
 
   test("Write and read an encrypted file") {
+    val conf = spark.sessionState.newHadoopConf()
+    val provider = HadoopShimsFactory.get.getHadoopKeyProvider(conf, new Random)
+    assume(!provider.getKeyNames.isEmpty,
+      s"$provider doesn't has the test keys. ORC shim is created with old Hadoop libraries")

Review comment:
       Just a question; `old` means v2.6 and earlier?




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on a change in pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on a change in pull request #31697:
URL: https://github.com/apache/spark/pull/31697#discussion_r585302235



##########
File path: sql/core/src/test/scala/org/apache/spark/sql/execution/datasources/orc/OrcEncryptionSuite.scala
##########
@@ -30,6 +32,11 @@ class OrcEncryptionSuite extends OrcTest with SharedSparkSession {
     Row(null, "841626795E7D351555B835A002E3BF10669DE9B81C95A3D59E10865AC37EA7C3", "Dongjoon Hyun")
 
   test("Write and read an encrypted file") {
+    val conf = spark.sessionState.newHadoopConf()
+    val provider = HadoopShimsFactory.get.getHadoopKeyProvider(conf, new Random)
+    assume(!provider.getKeyNames.isEmpty,
+      s"$provider doesn't has the test keys. ORC shim is created with old Hadoop libraries")

Review comment:
       Yes~ 2.6 and below here.




----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA removed a comment on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
SparkQA removed a comment on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788642908


   **[Test build #135625 has started](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135625/testReport)** for PR 31697 at commit [`27c87a5`](https://github.com/apache/spark/commit/27c87a56c55217e992669634577559bab246526f).


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788637944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40205/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon closed pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
HyukjinKwon closed pull request #31697:
URL: https://github.com/apache/spark/pull/31697


   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788961004


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135625/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788631084


   Hi, @HyukjinKwon and @maropu .
   Last time, I recovered GitHub Action and Jenkins PR Builder, but missed Jenkins Maven run.
   This PR generalizes this test suite by refactoring. Now, we can remove the newly added GitHub Action and preserve SBT test coverage (GitHub Action/Jenkins). Also, the Maven Jenkins will be recovered by ignoring the test failures.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun edited a comment on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun edited a comment on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788631084


   Hi, @HyukjinKwon and @maropu . Could you review this PR again?
   Last time, I recovered GitHub Action and Jenkins PR Builder, but missed Jenkins Maven run.
   This PR generalizes this test suite by refactoring. Now, we can remove the newly added GitHub Action and preserve SBT test coverage (GitHub Action/Jenkins). Also, the Maven Jenkins will be recovered by ignoring the test failures.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
AmplabJenkins commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788637944


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder-K8s/40205/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] HyukjinKwon commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
HyukjinKwon commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788697974


   Merged to master.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] SparkQA commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
SparkQA commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788932856


   **[Test build #135625 has finished](https://amplab.cs.berkeley.edu/jenkins/job/SparkPullRequestBuilder/135625/testReport)** for PR 31697 at commit [`27c87a5`](https://github.com/apache/spark/commit/27c87a56c55217e992669634577559bab246526f).
    * This patch passes all tests.
    * This patch merges cleanly.
    * This patch adds no public classes.


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] AmplabJenkins removed a comment on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
AmplabJenkins removed a comment on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788961004


   
   Refer to this link for build results (access rights to CI server needed): 
   https://amplab.cs.berkeley.edu/jenkins//job/SparkPullRequestBuilder/135625/
   


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788663396


   Thank you, @maropu and @HyukjinKwon .


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org

[GitHub] [spark] dongjoon-hyun commented on pull request #31697: [SPARK-34578][SQL][TESTS][test-maven] Refactor ORC encryption tests and ignore ORC shim loaded by old Hadoop library

Posted by GitBox <gi...@apache.org>.
dongjoon-hyun commented on pull request #31697:
URL: https://github.com/apache/spark/pull/31697#issuecomment-788858110


   Thank you!


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org