[jira] [Commented] (YARN-2468) Log handling for LRS

["Xuan Gong (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2468?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14114754#comment-14114754 ] 

Xuan Gong commented on YARN-2468:
---------------------------------

We will rely on user’s log application (such as log4j) to do the rollover for the logs. We have already exposed an environment variable LOG_DIR that the users can use to set up their log application.  We will provide a log service that will upload all the qualified logs periodically. 

Several changes in this patch:
1. Create a logContext (add into ApplicationSubmissionContext), which includes include_patterns, exclude_patterns (Those are used to filter the logs) and intervals (defines how often log aggregation service uploads the container logs).
2. AppLogAggregatorImpl will upload container logs periodically instead of waiting for the application to finish.
3. Change the log layout. Currently, all container logs (for the same NM) will be written into a single file. This does not work for LRS. We will create a directory (named as node id of the NM), under this directory, every time when AppLogAggregatorImpl starts to upload container logs; it will create a file (named as node_id + timestamp).  


> Log handling for LRS
> --------------------
>
>                 Key: YARN-2468
>                 URL: https://issues.apache.org/jira/browse/YARN-2468
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Xuan Gong
>            Assignee: Xuan Gong
>
> Currently, when application is finished, NM will start to do the log aggregation. But for Long running service applications, this is not ideal. The problems we have are:
> 1) LRS applications are expected to run for a long time (weeks, months).
> 2) Currently, all the container logs (from one NM) will be written into a single file. The files could become larger and larger.



--
This message was sent by Atlassian JIRA
(v6.2#6252)