[jira] [Commented] (YARN-221) NM should provide a way for AM to tell it not to aggregate logs.

["Ming Ma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-221?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14598663#comment-14598663 ] 

Ming Ma commented on YARN-221:
------------------------------

Thanks [~xgong]. How about the followings?

* Allow applications to specify the policy parameter via LogAggregationContext along with the policy class.

{noformat}
public abstract class LogAggregationContext {
    public void setContainerLogPolicyClass(Class<? extends ContainerLogAggregationPolicy> logPolicy);
    public Class<? extends ContainerLogAggregationPolicy> getContainerLogPolicyClass();
    public void setParameters(String parameters);
    public String getParameters();
}
{noformat}

* NM uses default cluster-wide settings via the following configurations. MR can override these configurations on per-application basis.

{noformat}
<property>
    <name>yarn.container-log-aggregation-policy.class</name>
    <value>org.apache.hadoop.yarn.container-log-aggregation-policy.SampleRateContainerLogAggregationPolicy</value>
</property>
<property>
    <name>yarn.container-log-aggregation-policy.class.SampleRateContainerLogAggregationPolicy</name>
    <value>SR:0.2</value>
</property>
{noformat}

* To support per-application policy, modify MR YarnRunner. We can also modify YarnClientImpl to read these configurations and set ApplicationSubmissionContext accordingly.

* The log aggregation policy object loaded in NM can be shared among different applications as long as they belong to same policy class with the same parameters.

> NM should provide a way for AM to tell it not to aggregate logs.
> ----------------------------------------------------------------
>
>                 Key: YARN-221
>                 URL: https://issues.apache.org/jira/browse/YARN-221
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: log-aggregation, nodemanager
>            Reporter: Robert Joseph Evans
>            Assignee: Ming Ma
>         Attachments: YARN-221-trunk-v1.patch, YARN-221-trunk-v2.patch, YARN-221-trunk-v3.patch, YARN-221-trunk-v4.patch, YARN-221-trunk-v5.patch
>
>
> The NodeManager should provide a way for an AM to tell it that either the logs should not be aggregated, that they should be aggregated with a high priority, or that they should be aggregated but with a lower priority.  The AM should be able to do this in the ContainerLaunch context to provide a default value, but should also be able to update the value when the container is released.
> This would allow for the NM to not aggregate logs in some cases, and avoid connection to the NN at all.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)