[jira] [Assigned] (BIGTOP-3093) Modifying the way Spark's log4j.properties file is being generated

["Evans Ye (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/BIGTOP-3093?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Evans Ye reassigned BIGTOP-3093:
--------------------------------

    Assignee: Amiya Chakraborty

> Modifying the way Spark's log4j.properties file is being generated
> ------------------------------------------------------------------
>
>                 Key: BIGTOP-3093
>                 URL: https://issues.apache.org/jira/browse/BIGTOP-3093
>             Project: Bigtop
>          Issue Type: Improvement
>          Components: spark
>            Reporter: Amiya Chakraborty
>            Assignee: Amiya Chakraborty
>            Priority: Major
>              Labels: patch
>         Attachments: SAMPLE patch-Modify_Spark_log4j.properties_file_generation.diff, patch-Modifying_the_way_Spark_log4j.properties_file_is_being_generated.diff
>
>   Original Estimate: 24h
>  Remaining Estimate: 24h
>
> *WHY ?*
> Currently, when Bigtop's spark puppet module is deployed, it takes /etc/spark/conf/log4j.properties.template (which comes from the Spark distribution) and generates /etc/spark/conf/log4j.properties.
> However, there is a disadvantage to this - any custom configuration overrides (originating from .yaml files) containing developer-friendly comments are not successfully parsed and subsequently, can't make to /etc/spark/conf/log4j.properties file.As a result, from troubleshooting/debugging viewpoint; down the line any support eng./developer would find it difficult to interpret significance of these settings.
> For example, the following log4j rolling settings would get added to generated /etc/spark/conf/log4j.properties file, but without the associated comments:
> _# Set the default spark streaming app log level to INFO. When running spark streaming app, # the logs generated by this class will be rolled on hourly basis to avoid deletion_ 
> _# of stdout and stderr logs by LogPusher on crossing the 4 hour retention period log4j.logger.org.apache.spark.streaming=INFO, DRFA-stderr, DRFA-stdout log4j.appender.DRFA-stderr=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA-stderr.file=${spark.yarn.app.container.log.dir}/stderr log4j.appender.DRFA-stderr.DatePattern='.'yyyy-MM-dd-HH_ 
> _log4j.appender.DRFA-stderr.layout=org.apache.log4j.PatternLayout_ 
> _log4j.appender.DRFA-stderr.layout.ConversionPattern=%d\{ISO8601} %p [%t] %c:%m%n_ 
> _#Roll stdout logs_ 
> _log4j.appender.DRFA-stdout=org.apache.log4j.DailyRollingFileAppender log4j.appender.DRFA-stdout.file=${spark.yarn.app.container.log.dir}/stdout log4j.appender.DRFA-stdout.DatePattern='.'yyyy-MM-dd-HH_ 
> _log4j.appender.DRFA-stdout.layout=org.apache.log4j.PatternLayout_ 
> _log4j.appender.DRFA-stdout.layout.ConversionPattern=%d\{ISO8601} %p [%t] %c:%m%n
> _Gradually, as the Spark's log4j.properties file will grow with numerous configurations; absence of comments will render log4j.properties file difficult to maintain. Besides, given that many Cloud vendors override Spark's log4j.properties configurations; this will be beneficial to substantial number of users. Please find enclosed sample patch file providing glimpse of how the "overrides" attribute is being used. 
> *OBJECTIVE *
> This Jira intends to expedite the process and get Spark's log4j.properties file be sourced on newly added template rather than being sourced on open source Spark distribution file log4j.properties.template.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)