You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Rafal Mucha (Jira)" <ji...@apache.org> on 2024/02/24 13:55:00 UTC
[jira] [Commented] (SPARK-44646) Migrate Log4j 2.x in Spark 3.4.1 to Logback
[ https://issues.apache.org/jira/browse/SPARK-44646?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17820343#comment-17820343 ]
Rafal Mucha commented on SPARK-44646:
-------------------------------------
Hi [~viirya] ,
Do you have any plans to somehow address this issue? It is quite serious problem, since it force using log4j2 implementation (log4j-core) with Spark. Solution is either quite naive approach like [here|https://github.com/apache/spark/pull/45001] , or on the other hand, creating separate interfaces in Spark core and developing packages that offer implementations utilising different logging frameworks - it is spring approach. Have you had any discussions about that?
Best
> Migrate Log4j 2.x in Spark 3.4.1 to Logback
> -------------------------------------------
>
> Key: SPARK-44646
> URL: https://issues.apache.org/jira/browse/SPARK-44646
> Project: Spark
> Issue Type: Brainstorming
> Components: Build
> Affects Versions: 3.4.1
> Reporter: Yu Tian
> Priority: Major
> Attachments: Screenshot 2023-08-09 at 2.40.12 PM.png
>
>
> Hi,
> We are working on the spark 3.4.1 upgrade from spark 3.1.3, in our logging system, we are using logback framework, it is working with spark 3.1.3 since it is using log4j 1.x. However, when we upgrade spark to 3.4.1, based on the [release notes|https://spark.apache.org/docs/latest/core-migration-guide.html], spark is migrating from log4j 2.x from log4j 1.x, the way we are replacing the log4j with logback is causing build failures in spark master start process.
> Error: Unable to initialize main class org.apache.spark.deploy.master.Master
> Caused by: java.lang.NoClassDefFoundError: org/apache/logging/log4j/core/Filter
> In our current approach, we are using log4j-over-slf4j to replace the log4j-core, it is only applicable to log4j 1.x library. And there is no log4j-over-slf4j for log4j 2.x out there yet. (please correct me if I am wrong).
> I am also curious that why spark choose to use log4j 2.x instead of using SPI, which gives the users less flexibility to choose whatever logger implementation they want to use.
> I want to share this issue and see if anyone else has been reported this and if there is any work-around or alternative solutions for it. Any suggestions are appreciated, thanks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org