[jira] [Commented] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.

["Ming Ma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13916958#comment-13916958 ] 

Ming Ma commented on YARN-221:
------------------------------

Thanks, Jason.

To fix the racing between container's exit by itself and MRAppMaster's stopContainer, I will upload the patch to https://issues.apache.org/jira/browse/MAPREDUCE-5465.

To support the feature of being able to have X% of container logs aggregated, we can do it at the yarn layer instead at MR layer. In that way, other applications can get it from free.

If AM doesn't specify any log aggregation policy as part of ContainerLaunchContext, yarn's default log aggregation policy will be applied. The default polices could be like:

For worker containers,
    1. Always aggregate logs of failed or killed containers.
    2. Aggregate subset of container logs. The sample rate is configurable and it is specific to the application.

For AM containers,
    1. Always aggregate logs of failed or killed containers.
    2. By default, AM log will be aggregated regardless of the status. It can be disabled via configuration and that will only impact succeeded container.

Comments?

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Chris Trezzo
>         Attachments: YARN-221-trunk-v1.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the logs should not be aggregated, that they should be aggregated with a high priority, or that they should be aggregated but with a lower priority.  The AM should be able to do this in the ContainerLaunch context to provide a default value, but should also be able to update the value when the container is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid connection to the NN at all.



--
This message was sent by Atlassian JIRA
(v6.1.5#6160)