[jira] [Created] (BIGTOP-3093) Modifying the way Spark's log4j.properties file is being generated

["Amiya Chakraborty (JIRA)" <ji...@apache.org>]

Amiya Chakraborty created BIGTOP-3093:
-----------------------------------------

             Summary: Modifying the way Spark's log4j.properties file is being generated
                 Key: BIGTOP-3093
                 URL: https://issues.apache.org/jira/browse/BIGTOP-3093
             Project: Bigtop
          Issue Type: Improvement
          Components: spark
            Reporter: Amiya Chakraborty
         Attachments: SAMPLE patch-Modify_Spark_log4j.properties_file_generation.diff

*WHY ?*

Currently, when Bigtop's spark puppet module is deployed, it takes /etc/spark/conf/log4j.properties.template (which comes from the Spark distribution) and generates /etc/spark/conf/log4j.properties.

However, there is a disadvantage to this - any custom configuration overrides (originating from .yaml files) containing developer-friendly comments are not successfully parsed and subsequently, can't make to /etc/spark/conf/log4j.properties file.As a result, from troubleshooting/debugging viewpoint; down the line any support eng./developer would find it difficult to interpret significance of these settings.

For example, the following log4j rolling settings would get added to generated /etc/spark/conf/log4j.properties file, but without the associated comments:



_# Set the default spark streaming app log level to INFO. When running spark streaming app, # the logs generated by this class will be rolled on hourly basis to avoid deletion_ 
_# of stdout and stderr logs by LogPusher on crossing the 4 hour retention period log4j.logger.org.apache.spark.streaming=INFO, DRFA-stderr, DRFA-stdout log4j.appender.DRFA-stderr=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA-stderr.file=${spark.yarn.app.container.log.dir}/stderr log4j.appender.DRFA-stderr.DatePattern='.'yyyy-MM-dd-HH_ 
_log4j.appender.DRFA-stderr.layout=org.apache.log4j.PatternLayout_ 
_log4j.appender.DRFA-stderr.layout.ConversionPattern=%d\{ISO8601} %p [%t] %c:%m%n_ 
_#Roll stdout logs_ 
_log4j.appender.DRFA-stdout=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA-stdout.file=${spark.yarn.app.container.log.dir}/stdout log4j.appender.DRFA-stdout.DatePattern='.'yyyy-MM-dd-HH_ 
_log4j.appender.DRFA-stdout.layout=org.apache.log4j.PatternLayout_ 
_log4j.appender.DRFA-stdout.layout.ConversionPattern=%d\{ISO8601} %p [%t] %c:%m%n

_Gradually, as the Spark's log4j.properties file will grow with numerous configurations; absence of comments will render log4j.properties file difficult to maintain. Besides, given that many Cloud vendors override Spark's log4j.properties configurations; this will be beneficial to substantial number of users. Please find enclosed sample patch file providing glimpse of how the "overrides" attribute is being used. 

OBJECTIVE ?

This Jira intends to expedite the process and get Spark's log4j.properties file be sourced on newly added template rather than being sourced on open source Spark distribution file log4j.properties.template.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)