You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@spark.apache.org by GitBox <gi...@apache.org> on 2022/03/11 01:49:28 UTC
[GitHub] [spark] wangyum opened a new pull request #35811: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
wangyum opened a new pull request #35811:
URL: https://github.com/apache/spark/pull/35811
### What changes were proposed in this pull request?
Add `log4j-core` and `log4j-api` to classpath if active hadoop-provided.
### Why are the changes needed?
`log4j-core` is needed:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter
at java.lang.Class.getDeclaredMethods0(Native Method)
at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
at java.lang.Class.getMethod0(Class.java:3018)
at java.lang.Class.getMethod(Class.java:1784)
at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.Filter
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 7 more
```
`log4j-api` is needed:
```
Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
at org.apache.spark.deploy.yarn.SparkRackResolver.<init>(SparkRackResolver.scala:42)
at org.apache.spark.deploy.yarn.SparkRackResolver$.get(SparkRackResolver.scala:114)
at org.apache.spark.scheduler.cluster.YarnScheduler.<init>(YarnScheduler.scala:31)
at org.apache.spark.scheduler.cluster.YarnClusterManager.createTaskScheduler(YarnClusterManager.scala:35)
at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2985)
at org.apache.spark.SparkContext.<init>(SparkContext.scala:563)
at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
at scala.Option.getOrElse(Option.scala:189)
at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:327)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:159)
at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
at java.lang.reflect.Method.invoke(Method.java:498)
at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.LogManager
at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
... 26 more
```
### Does this PR introduce _any_ user-facing change?
No.
### How was this patch tested?
manual test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eejbyfeldt commented on pull request #35811: [SPARK-38516][BUILD] Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
eejbyfeldt commented on pull request #35811:
URL: https://github.com/apache/spark/pull/35811#issuecomment-1064855439
I build this branch with ` ./dev/make-distribution.sh --tgz --name hadoop-provided-test -Phadoop-provided -Pyarn` and then run `org.apache.spark.examples.SparkPi` example with hadoop 3.3.2. That example job runs fine but I do get slf4j warnigns regarding multiple bindings:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:../spark-3.3.0-SNAPSHOT-bin-hadoop-provided-test/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:../hadoop-3.3.2/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #35811: [SPARK-38516][BUILD] Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #35811:
URL: https://github.com/apache/spark/pull/35811#discussion_r824548662
##########
File path: pom.xml
##########
@@ -744,13 +744,11 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<!-- API bridge between log4j 1 and 2 -->
Review comment:
Thank you @eejbyfeldt . Removed it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eejbyfeldt commented on a change in pull request #35811: [SPARK-38516][BUILD] Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
eejbyfeldt commented on a change in pull request #35811:
URL: https://github.com/apache/spark/pull/35811#discussion_r824441317
##########
File path: pom.xml
##########
@@ -744,13 +744,11 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<!-- API bridge between log4j 1 and 2 -->
Review comment:
If we include `lof4j-slf4j-impl` will we then not end up with multiple slf4j bindings on the classpath since hadoop already includes `slf4j-log4j12`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eejbyfeldt edited a comment on pull request #35811: [SPARK-38516][BUILD] Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
eejbyfeldt edited a comment on pull request #35811:
URL: https://github.com/apache/spark/pull/35811#issuecomment-1064855439
I built this branch with ` ./dev/make-distribution.sh --tgz --name hadoop-provided-test -Phadoop-provided -Pyarn` and then run `org.apache.spark.examples.SparkPi` example with hadoop 3.3.2. That example job runs fine but I do get slf4j warnings regarding multiple bindings:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:../spark-3.3.0-SNAPSHOT-bin-hadoop-provided-test/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:../hadoop-3.3.2/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] eejbyfeldt edited a comment on pull request #35811: [SPARK-38516][BUILD] Add log4j-core, log4j-api and log4j-slf4j-impl to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
eejbyfeldt edited a comment on pull request #35811:
URL: https://github.com/apache/spark/pull/35811#issuecomment-1064855439
I built this branch with ` ./dev/make-distribution.sh --tgz --name hadoop-provided-test -Phadoop-provided -Pyarn` and then run `org.apache.spark.examples.SparkPi` example with hadoop 3.3.2. That example job runs fine but I do get slf4j warnigns regarding multiple bindings:
```
SLF4J: Class path contains multiple SLF4J bindings.
SLF4J: Found binding in [jar:file:../spark-3.3.0-SNAPSHOT-bin-hadoop-provided-test/jars/log4j-slf4j-impl-2.17.1.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: Found binding in [jar:file:../hadoop-3.3.2/share/hadoop/common/lib/slf4j-log4j12-1.7.30.jar!/org/slf4j/impl/StaticLoggerBinder.class]
SLF4J: See http://www.slf4j.org/codes.html#multiple_bindings for an explanation.
SLF4J: Actual binding is of type [org.apache.logging.slf4j.Log4jLoggerFactory]
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35811: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35811:
URL: https://github.com/apache/spark/pull/35811#discussion_r824331185
##########
File path: pom.xml
##########
@@ -744,13 +744,11 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<!-- API bridge between log4j 1 and 2 -->
Review comment:
Then I think `log4j-slf4j-impl` is also needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] dongjoon-hyun closed pull request #35811: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
dongjoon-hyun closed pull request #35811:
URL: https://github.com/apache/spark/pull/35811
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] viirya commented on a change in pull request #35811: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
viirya commented on a change in pull request #35811:
URL: https://github.com/apache/spark/pull/35811#discussion_r824331185
##########
File path: pom.xml
##########
@@ -744,13 +744,11 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<!-- API bridge between log4j 1 and 2 -->
Review comment:
Then I think `log4j-slf4j-impl` and `log4j-1.2-api` are also needed?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org
[GitHub] [spark] wangyum commented on a change in pull request #35811: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
Posted by GitBox <gi...@apache.org>.
wangyum commented on a change in pull request #35811:
URL: https://github.com/apache/spark/pull/35811#discussion_r824358899
##########
File path: pom.xml
##########
@@ -744,13 +744,11 @@
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-api</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<groupId>org.apache.logging.log4j</groupId>
<artifactId>log4j-core</artifactId>
<version>${log4j.version}</version>
- <scope>${hadoop.deps.scope}</scope>
</dependency>
<dependency>
<!-- API bridge between log4j 1 and 2 -->
Review comment:
Yes. log4j-slf4j-impl is also needed:
https://github.com/apache/spark/blob/1431a4aa74cf69b3ee607e313dece6fed8390de6/core/src/main/scala/org/apache/spark/internal/Logging.scala#L228-L234
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
---------------------------------------------------------------------
To unsubscribe, e-mail: reviews-unsubscribe@spark.apache.org
For additional commands, e-mail: reviews-help@spark.apache.org