You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@spark.apache.org by "Hari Sekhon (JIRA)" <ji...@apache.org> on 2018/06/26 15:48:01 UTC
[jira] [Comment Edited] (SPARK-6305) Add support for log4j 2.x to
Spark
[ https://issues.apache.org/jira/browse/SPARK-6305?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16523888#comment-16523888 ]
Hari Sekhon edited comment on SPARK-6305 at 6/26/18 3:47 PM:
-------------------------------------------------------------
Log4j 2.x would really help with Spark logging integration to ELK as there are a lot of things that just don't work properly in Log4j 1.x like layout.ConversionPattern for constructing JSON enriched logs, such as logging user and app names to distinguish jobs and provide much needed search usability. This is simply ignored in the SocketAppender in Log4j 1.x, while SyslogAppender respects ConversionPattern but then splits all Java Exceptions in to multiple syslog logs so the JSON no longer parses and routes to the right indices for the Yarn queue, nor can you reassemble the exception logs using multiline codec at the other end as you'd end up with corrupted input streams from multiple loggers) :-/
Running Filebeats everywhere instead seems like overkill compared to being able to enable logging for debugging jobs on an ad-hoc basis to a Logstash sink that works using much better Log4j 2.x output appenders.
I hope someone finally manages to sort this out as it's years overdue given Log4j 1.x was end of life 3 years ago and there is a big jump in capabilities between Log4j 1.x and 2.x, both in the number of appenders as well as completeness of even the old appenders such as the SocketAppender as mentioned above.
was (Author: harisekhon):
Log4j 2.x would really help with Spark logging integration to ELK as there are a lot of things that just don't work properly in Log4j 1.x like layout.ConversionPattern for constructing JSON enriched logs, such as logging user and app names to distinguish jobs and provide much needed search usability. This is simply ignored in the SocketAppender in Log4j 1.x :-/ while SyslogAppender respects ConversionPattern but then splits all Java Exceptions in to multiple syslog logs so the JSON no longer parses and routes to the right indices for the Yarn queue, nor can you reassemble the exception logs using multiline codec at the other end as you'd end up with corrupted input streams from multiple loggers) :-/
Running Filebeats everywhere instead seems like overkill compared to being able to enable logging for debugging jobs on an ad-hoc basis to a Logstash sink that works using much better Log4j 2.x output appenders.
I hope someone finally manages to sort this out as it's years overdue given Log4j 1.x was end of life 3 years ago and there is a big jump in capabilities between Log4j 1.x and 2.x, both in the number of appenders as well as completeness of even the old appenders such as the SocketAppender as mentioned above.
> Add support for log4j 2.x to Spark
> ----------------------------------
>
> Key: SPARK-6305
> URL: https://issues.apache.org/jira/browse/SPARK-6305
> Project: Spark
> Issue Type: Improvement
> Components: Build
> Reporter: Tal Sliwowicz
> Priority: Minor
>
> log4j 2 requires replacing the slf4j binding and adding the log4j jars in the classpath. Since there are shaded jars, it must be done during the build.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@spark.apache.org
For additional commands, e-mail: issues-help@spark.apache.org