You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2021/09/16 14:26:08 UTC
[GitHub] [spark] LuciferYang commented on a change in pull request #30701: [SPARK-33212][BUILD] Upgrade to Hadoop 3.2.2 and move to shaded clients for Hadoop 3.x profile
LuciferYang commented on a change in pull request #30701:
URL: https://github.com/apache/spark/pull/30701#discussion_r710172178
##########
File path: core/pom.xml
##########
@@ -66,7 +66,13 @@
</dependency>
<dependency>
<groupId>org.apache.hadoop</groupId>
- <artifactId>hadoop-client</artifactId>
+ <artifactId>${hadoop-client-api.artifact}</artifactId>
Review comment:
@sunchao @dongjoon-hyun I find there are some bad case with using variables here:
- if we run `mvn dependency:tree -pl resource-managers/yarn -Phadoop-2.7 -Pyarn -am`, the dependency-tree is correct
- if we run `mvn dependency:tree -pl resource-managers/yarn -Phadoop-2.7 -Pyarn`, the dependency-tree is wrong, the dependency of hadoop 3.3.1 exists in the dependency tree, even though we have added the profile of - Phadoop2.7
So if we execute
```
mvn clean install -DskipTests -pl resource-managers/yarn -am -Phadoop-2.7 -Pyarn
mvn test -pl resource-managers/yarn -Phadoop-2.7 -Pyarn -DwildcardSuites=org.apache.spark.deploy.yarn.YarnClusterSuite
```
The test case `YarnClusterSuite` will failed as follows:
```
Discovery starting.
Discovery completed in 277 milliseconds.
Run starting. Expected test count is: 1
YarnClusterSuite:
*** RUN ABORTED ***
java.lang.NoClassDefFoundError: org/apache/hadoop/shaded/com/google/inject/servlet/ServletModule
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
at java.net.URLClassLoader$1.run(URLClassLoader.java:363)
at java.security.AccessController.doPrivileged(Native Method)
at java.net.URLClassLoader.findClass(URLClassLoader.java:362)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
...
Cause: java.lang.ClassNotFoundException: org.apache.hadoop.shaded.com.google.inject.servlet.ServletModule
at java.net.URLClassLoader.findClass(URLClassLoader.java:382)
at java.lang.ClassLoader.loadClass(ClassLoader.java:419)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:352)
at java.lang.ClassLoader.loadClass(ClassLoader.java:352)
at java.lang.ClassLoader.defineClass1(Native Method)
at java.lang.ClassLoader.defineClass(ClassLoader.java:757)
at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:142)
at java.net.URLClassLoader.defineClass(URLClassLoader.java:468)
at java.net.URLClassLoader.access$100(URLClassLoader.java:74)
at java.net.URLClassLoader$1.run(URLClassLoader.java:369)
...
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org