You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@spark.apache.org by gu...@apache.org on 2021/07/09 06:19:59 UTC
[spark] branch branch-3.2 updated: [SPARK-36067][BUILD][TEST][YARN]
YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2
profile is activated explicitly
This is an automated email from the ASF dual-hosted git repository.
gurwls223 pushed a commit to branch branch-3.2
in repository https://gitbox.apache.org/repos/asf/spark.git
The following commit(s) were added to refs/heads/branch-3.2 by this push:
new 3c825a1 [SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
3c825a1 is described below
commit 3c825a18b64d3330e01f2fa62604771b2019e5e6
Author: Kousuke Saruta <sa...@oss.nttdata.com>
AuthorDate: Fri Jul 9 15:18:52 2021 +0900
[SPARK-36067][BUILD][TEST][YARN] YarnClusterSuite fails due to NoClassDefFoundError unless hadoop-3.2 profile is activated explicitly
### What changes were proposed in this pull request?
This PR fixes an issue that `YarnClusterSuite` fails due to `NoClassDefFoundError unless `hadoop-3.2` profile is activated explicitly regardless of building with SBT or Maven.
```
build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite"
...
[info] YarnClusterSuite:
[info] org.apache.spark.deploy.yarn.YarnClusterSuite *** ABORTED *** (598 milliseconds)
[info] java.lang.NoClassDefFoundError: org/bouncycastle/operator/OperatorCreationException
[info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager$RMActiveServices.serviceInit(ResourceManager.java:888)
[info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
[info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.createAndInitActiveServices(ResourceManager.java:1410)
[info] at org.apache.hadoop.yarn.server.resourcemanager.ResourceManager.serviceInit(ResourceManager.java:344)
[info] at org.apache.hadoop.service.AbstractService.init(AbstractService.java:164)
[info] at org.apache.hadoop.yarn.server.MiniYARNCluster.initResourceManager(MiniYARNCluster.java:359)
```
The solution is modifying `yarn/pom.xml` to activate `hadoop-3.2` profiles by default.
### Why are the changes needed?
hadoop-3.2 profile should be enabled by default so `YarnClusterSuite` should also successfully finishes without `-Phadoop-3.2`.
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
Run `YarnClusterSuite` with both SBT and Maven without `-Phadoop-3.2` and it successfully finished.
```
build/sbt -Pyarn "yarn/testOnly org.apache.spark.deploy.yarn.YarnClusterSuite"
...
[info] Run completed in 5 minutes, 38 seconds.
[info] Total number of tests run: 27
[info] Suites: completed 1, aborted 0
[info] Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0
[info] All tests passed.
build/mvn -Pyarn -pl resource-managers/yarn test -Dtest=none -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite
...
Run completed in 5 minutes, 49 seconds.
Total number of tests run: 27
Suites: completed 2, aborted 0
Tests: succeeded 27, failed 0, canceled 0, ignored 0, pending 0
All tests passed.
```
Closes #33276 from sarutak/fix-bouncy-issue.
Authored-by: Kousuke Saruta <sa...@oss.nttdata.com>
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
(cherry picked from commit 9aa65a6e0efbe9d12670ce655cbf0119fc2d3173)
Signed-off-by: Hyukjin Kwon <gu...@apache.org>
---
resource-managers/yarn/pom.xml | 3 +++
1 file changed, 3 insertions(+)
diff --git a/resource-managers/yarn/pom.xml b/resource-managers/yarn/pom.xml
index 00ad7e8..e5fef2a 100644
--- a/resource-managers/yarn/pom.xml
+++ b/resource-managers/yarn/pom.xml
@@ -78,6 +78,9 @@
</profile>
<profile>
<id>hadoop-3.2</id>
+ <activation>
+ <activeByDefault>true</activeByDefault>
+ </activation>
<dependencies>
<!-- Used by MiniYARNCluster -->
<dependency>
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org