[jira] [Commented] (MAPREDUCE-5672) Provide optional RollingFileAppender for container log4j (syslog)

["Jason Lowe (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-5672?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13864453#comment-13864453 ] 

Jason Lowe commented on MAPREDUCE-5672:
---------------------------------------

Thanks for updating the patch, looks a lot cleaner now that it's focused on just the RFA feature.

As for the configuration property names, I think your proposed names are fine.  However it does bring up the question I should have asked earlier as to whether they really need to be separate settings.  Will one want to set the AM to be rolling but not the tasks or vice-versa?  Having two properties to configure is worse than just one if there's no real reason to configure them differently.  Or if there is a legitimate reason then one might wonder why tasks aren't separated into maps vs. reduces so they could be configured independently as well.  I'd prefer there to be simply one property to set, but I'm wondering if there's a use-case for having them separate that I'm missing.

> Provide optional RollingFileAppender for container log4j (syslog)
> -----------------------------------------------------------------
>
>                 Key: MAPREDUCE-5672
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-5672
>             Project: Hadoop Map/Reduce
>          Issue Type: Improvement
>          Components: mr-am, mrv2
>    Affects Versions: 2.2.0
>            Reporter: Gera Shegalov
>            Assignee: Gera Shegalov
>         Attachments: MAPREDUCE-5672.v01.patch, MAPREDUCE-5672.v02.patch, MAPREDUCE-5672.v03.patch, MAPREDUCE-5672.v04.patch, MAPREDUCE-5672.v05.patch, Screen Shot 2013-12-05 at 3.21.02 PM.png, Screen Shot 2013-12-05 at 3.23.33 PM.png
>
>
> This JIRA is an alternative take on YARN-1130
> We propose providing an option of using a RollingFileAppender(RFA)-based implementation of container log appender as means of log size control via mapreduce.task.userlog.limit.kb. 
> The idea is to use mapreduce.task.userlog.limit.kb as maximumFileSize of RFA. In addition yarn.app.mapreduce.container.log.backups (task attempt containers) and yarn.app.mapreduce.am.log.backups (MR-AM) are passed as maxBackupIndex.
> Both current ContainerLogAppender (CLA) and new ContainerRollingLogAppender (CRLA) co-exist. CLA is the default. CRLA is chosen when  mapreduce.task.userlog.limit.kb > 0 && *.backups > 0.
> Pros: 
> 1) CRLA output is visible in UI right away. CLA output with mapreduce.task.userlog.limit.kb > 0 is not visible until the task attempt finishes that prevents timely diagnostics. 
> 2) Even with excessive logging and a large mapreduce.task.userlog.limit.kb, no space is taken from the JVM heap.
> 3) No UI impact, since YARN is already designed to deal with any log name beyond stderr/out, syslog, debug.out, profile.out
> Cons:
> 1) if the logging is excessive there will be more local filesystem metadata I/O due to roll. That should be negligible in the grand scheme.
> Furthermore, to improve log consistency and completeness in the case of JVM crashes and SIGTERMing by NM, we propose to restore the MRv1 behavior of periodic log syncing (every 5s) and having log sync as part of a shutdown hook.
>  



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)