You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by do...@apache.org on 2023/03/27 16:43:02 UTC
[spark] branch branch-3.4 updated: [SPARK-42934][BUILD] Add `spark.hadoop.hadoop.security.key.provider.path` to `scalatest-maven-plugin`
This is an automated email from the ASF dual-hosted git repository.
dongjoon pushed a commit to branch branch-3.4
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.4 by this push:
new d7f2a6bc1af [SPARK-42934][BUILD] Add `spark.hadoop.hadoop.security.key.provider.path` to `scalatest-maven-plugin`
d7f2a6bc1af is described below
commit d7f2a6bc1af136b61514c6ff50cfc86d57378657
Author: yangjie01 <ya...@baidu.com>
AuthorDate: Mon Mar 27 09:42:40 2023 -0700
[SPARK-42934][BUILD] Add `spark.hadoop.hadoop.security.key.provider.path` to `scalatest-maven-plugin`
### What changes were proposed in this pull request?
When testing `OrcEncryptionSuite` using maven, all test suites are always skipped. So this pr add `spark.hadoop.hadoop.security.key.provider.path` to `systemProperties` of `scalatest-maven-plugin` to make `OrcEncryptionSuite` can test by maven.
### Why are the changes needed?
Make `OrcEncryptionSuite` can test by maven.
### Does this PR introduce _any_ user-facing change?
No, just for maven test
### How was this patch tested?
- Pass GitHub Actions
- Manual testingļ¼
run
```
build/mvn clean install -pl sql/core -DskipTests -am
build/mvn test -pl sql/core -Dtest=none -DwildcardSuites=org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite
```
**Before**
```
Discovery starting.
Discovery completed in 3 seconds, 218 milliseconds.
Run starting. Expected test count is: 4
OrcEncryptionSuite:
21:57:58.344 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- Write and read an encrypted file !!! CANCELED !!!
[] was empty org.apache.orc.impl.NullKeyProvider5af5d76f doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:37)
- Write and read an encrypted table !!! CANCELED !!!
[] was empty org.apache.orc.impl.NullKeyProvider5ad6cc21 doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:65)
- SPARK-35325: Write and read encrypted nested columns !!! CANCELED !!!
[] was empty org.apache.orc.impl.NullKeyProvider691124ee doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:116)
- SPARK-35992: Write and read fully-encrypted columns with default masking !!! CANCELED !!!
[] was empty org.apache.orc.impl.NullKeyProvider5403799b doesn't has the test keys. ORC shim is created with old Hadoop libraries (OrcEncryptionSuite.scala:166)
21:58:00.035 WARN org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite:
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.orc.OrcEncryptionSuite, threads: rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) =====
Run completed in 5 seconds, 41 milliseconds.
Total number of tests run: 0
Suites: completed 2, aborted 0
Tests: succeeded 0, failed 0, canceled 4, ignored 0, pending 0
No tests were executed.
```
**After**
```
Discovery starting.
Discovery completed in 3 seconds, 185 milliseconds.
Run starting. Expected test count is: 4
OrcEncryptionSuite:
21:58:46.540 WARN org.apache.hadoop.util.NativeCodeLoader: Unable to load native-hadoop library for your platform... using builtin-java classes where applicable
- Write and read an encrypted file
- Write and read an encrypted table
- SPARK-35325: Write and read encrypted nested columns
- SPARK-35992: Write and read fully-encrypted columns with default masking
21:58:51.933 WARN org.apache.spark.sql.execution.datasources.orc.OrcEncryptionSuite:
===== POSSIBLE THREAD LEAK IN SUITE o.a.s.sql.execution.datasources.orc.OrcEncryptionSuite, threads: rpc-boss-3-1 (daemon=true), shuffle-boss-6-1 (daemon=true) =====
Run completed in 8 seconds, 708 milliseconds.
Total number of tests run: 4
Suites: completed 2, aborted 0
Tests: succeeded 4, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
Closes #40566 from LuciferYang/SPARK-42934-2.
Authored-by: yangjie01 <ya...@baidu.com>
Signed-off-by: Dongjoon Hyun <do...@apache.org>
(cherry picked from commit a3d9e0ae0f95a55766078da5d0bf0f74f3c3cfc3)
Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
pom.xml | 2 ++
1 file changed, 2 insertions(+)
diff --git a/pom.xml b/pom.xml
index 6af47dd0eaf..18563a559ea 100644
--- a/pom.xml
+++ b/pom.xml
@@ -3027,6 +3027,8 @@
<spark.test.docker.removePulledImage>${spark.test.docker.removePulledImage}</spark.test.docker.removePulledImage>
<!-- Needed by sql/hive tests. -->
<test.src.tables>__not_used__</test.src.tables>
+ <!--SPARK-42934: Need by `OrcEncryptionSuite` -->
+ <spark.hadoop.hadoop.security.key.provider.path>test:///</spark.hadoop.hadoop.security.key.provider.path>
</systemProperties>
<tagsToExclude>${test.exclude.tags},${test.default.exclude.tags}</tagsToExclude>
<tagsToInclude>${test.include.tags}</tagsToInclude>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org