[spark] branch master updated: [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided

[do...@apache.org]

This is an automated email from the ASF dual-hosted git repository.

dongjoon pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/spark.git


The following commit(s) were added to refs/heads/master by this push:
     new 60334d7  [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
60334d7 is described below

commit 60334d7661b406a482c76970ecc43a9715996744
Author: Yuming Wang <yu...@ebay.com>
AuthorDate: Fri Mar 11 21:36:12 2022 -0800

    [SPARK-38516][BUILD] Add log4j-core and log4j-api to classpath if active hadoop-provided
    
    ### What changes were proposed in this pull request?
    
    Add `log4j-core` and `log4j-api` to classpath if active hadoop-provided.
    
    ### Why are the changes needed?
    
    `log4j-core` is needed:
    ```
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter
    	at java.lang.Class.getDeclaredMethods0(Native Method)
    	at java.lang.Class.privateGetDeclaredMethods(Class.java:2701)
    	at java.lang.Class.privateGetMethodRecursive(Class.java:3048)
    	at java.lang.Class.getMethod0(Class.java:3018)
    	at java.lang.Class.getMethod(Class.java:1784)
    	at sun.launcher.LauncherHelper.validateMainClass(LauncherHelper.java:544)
    	at sun.launcher.LauncherHelper.checkAndLoadMain(LauncherHelper.java:526)
    Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.core.Filter
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 7 more
    ```
    
    `log4j-api` is needed:
    ```
    Exception in thread "main" java.lang.NoClassDefFoundError: org/apache/logging/log4j/LogManager
    	at org.apache.spark.deploy.yarn.SparkRackResolver.<init>(SparkRackResolver.scala:42)
    	at org.apache.spark.deploy.yarn.SparkRackResolver$.get(SparkRackResolver.scala:114)
    	at org.apache.spark.scheduler.cluster.YarnScheduler.<init>(YarnScheduler.scala:31)
    	at org.apache.spark.scheduler.cluster.YarnClusterManager.createTaskScheduler(YarnClusterManager.scala:35)
    	at org.apache.spark.SparkContext$.org$apache$spark$SparkContext$$createTaskScheduler(SparkContext.scala:2985)
    	at org.apache.spark.SparkContext.<init>(SparkContext.scala:563)
    	at org.apache.spark.SparkContext$.getOrCreate(SparkContext.scala:2704)
    	at org.apache.spark.sql.SparkSession$Builder.$anonfun$getOrCreate$2(SparkSession.scala:953)
    	at scala.Option.getOrElse(Option.scala:189)
    	at org.apache.spark.sql.SparkSession$Builder.getOrCreate(SparkSession.scala:947)
    	at org.apache.spark.sql.hive.thriftserver.SparkSQLEnv$.init(SparkSQLEnv.scala:54)
    	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.<init>(SparkSQLCLIDriver.scala:327)
    	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver$.main(SparkSQLCLIDriver.scala:159)
    	at org.apache.spark.sql.hive.thriftserver.SparkSQLCLIDriver.main(SparkSQLCLIDriver.scala)
    	at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    	at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    	at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    	at java.lang.reflect.Method.invoke(Method.java:498)
    	at org.apache.spark.deploy.JavaMainApplication.start(SparkApplication.scala:52)
    	at org.apache.spark.deploy.SparkSubmit.org$apache$spark$deploy$SparkSubmit$$runMain(SparkSubmit.scala:958)
    	at org.apache.spark.deploy.SparkSubmit.doRunMain$1(SparkSubmit.scala:180)
    	at org.apache.spark.deploy.SparkSubmit.submit(SparkSubmit.scala:203)
    	at org.apache.spark.deploy.SparkSubmit.doSubmit(SparkSubmit.scala:90)
    	at org.apache.spark.deploy.SparkSubmit$$anon$2.doSubmit(SparkSubmit.scala:1046)
    	at org.apache.spark.deploy.SparkSubmit$.main(SparkSubmit.scala:1055)
    	at org.apache.spark.deploy.SparkSubmit.main(SparkSubmit.scala)
    Caused by: java.lang.ClassNotFoundException: org.apache.logging.log4j.LogManager
    	at java.net.URLClassLoader.findClass(URLClassLoader.java:381)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:424)
    	at sun.misc.Launcher$AppClassLoader.loadClass(Launcher.java:331)
    	at java.lang.ClassLoader.loadClass(ClassLoader.java:357)
    	... 26 more
    ```
    `log4j-slf4j-impl` is not needed: https://github.com/apache/spark/pull/35811#issuecomment-1064855439
    
    ### Does this PR introduce _any_ user-facing change?
    
    No.
    
    ### How was this patch tested?
    
    Manual test:
    ```shell
     ./dev/make-distribution.sh --name SPARK-38516 --tgz  -Phive -Phive-thriftserver  -Pyarn -Phadoop-2 -Phadoop-provided
    ```
    
    Closes #35811 from wangyum/SPARK-38516.
    
    Lead-authored-by: Yuming Wang <yu...@ebay.com>
    Co-authored-by: Yuming Wang <wg...@gmail.com>
    Signed-off-by: Dongjoon Hyun <do...@apache.org>
---
 pom.xml | 2 --
 1 file changed, 2 deletions(-)

diff --git a/pom.xml b/pom.xml
index 2c9ee34..5bc8d12 100644
--- a/pom.xml
+++ b/pom.xml
@@ -745,13 +745,11 @@
         <groupId>org.apache.logging.log4j</groupId>
         <artifactId>log4j-api</artifactId>
         <version>${log4j.version}</version>
-        <scope>${hadoop.deps.scope}</scope>
       </dependency>
       <dependency>
         <groupId>org.apache.logging.log4j</groupId>
         <artifactId>log4j-core</artifactId>
         <version>${log4j.version}</version>
-        <scope>${hadoop.deps.scope}</scope>
       </dependency>
       <dependency>
         <!-- API bridge between log4j 1 and 2 -->

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@spark.apache.org
For additional commands, e-mail: commits-help@spark.apache.org