You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "zhengchenyu (JIRA)" <ji...@apache.org> on 2017/07/03 02:59:01 UTC
[jira] [Comment Edited] (YARN-6396) Call
verifyAndCreateRemoteLogDir at service initialization instead of
application initialization to decrease load for name node
[ https://issues.apache.org/jira/browse/YARN-6396?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16071869#comment-16071869 ]
zhengchenyu edited comment on YARN-6396 at 7/3/17 2:58 AM:
-----------------------------------------------------------
The invocation 'createAppDir' in initAppAggregator will also visit the defaultNs! In our experience, it decreases the speed of application.
My issue "YARN-6728" seems a duplicate of this. Let's solve this problem together!
was (Author: zhengchenyu):
The invocation 'createAppDir' in initAppAggregator will also visit the defaultNs! In our experience, it decreases the speed of application.
> Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node
> -------------------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-6396
> URL: https://issues.apache.org/jira/browse/YARN-6396
> Project: Hadoop YARN
> Issue Type: Improvement
> Components: log-aggregation
> Affects Versions: 3.0.0-alpha2
> Reporter: zhihai xu
> Assignee: zhihai xu
> Priority: Minor
> Attachments: YARN-6396.000.patch
>
>
> Call verifyAndCreateRemoteLogDir at service initialization instead of application initialization to decrease load for name node.
> Currently for every application at each Node, verifyAndCreateRemoteLogDir will be called before doing log aggregation, This will be a non trivial overhead for name node in a large cluster since verifyAndCreateRemoteLogDir calls getFileStatus. Once the remote log directory is created successfully, it is not necessary to call it again. It will be better to call verifyAndCreateRemoteLogDir at LogAggregationService service initialization.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org