You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Tao Yang (JIRA)" <ji...@apache.org> on 2018/11/02 08:42:00 UTC
[jira] [Comment Edited] (YARN-8925) Updating distributed node
attributes only when necessary
[ https://issues.apache.org/jira/browse/YARN-8925?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16672769#comment-16672769 ]
Tao Yang edited comment on YARN-8925 at 11/2/18 8:41 AM:
---------------------------------------------------------
Hi, [~cheersyang]. I have attached V2 patch for review, this patch is not finished yet since some test cases are still needed, can you help to take a simple review and give your feedback about this patch?
V2 patch supports reporting node attributes only when updated or re-sync time is elapsed by NM to avoid overhead of the comparing in every heartbeat process. As your advice, logic is similar as reporting node labels. Common logic such as judging whether labels/attributes should be send to RM is extracted as HeartbeatSyncIfNeededHandler, so that handlers of labels and attributes can extend it to reuse the logic. For test cases, I have created TestNodeStatusUpdaterForAttributes to test NM side and will add some new test methods in TestResourceTrackerService to test RM side (similar to node labels test cases in TestResourceTrackerService). Any problems about that?
Looking forward to your feedback. Thanks !
was (Author: tao yang):
Hi, [~cheersyang]. I have attached V2 patch for review, this patch is not finished yet since some test cases are still needed, can you help to take a simple review and give your feedback about this patch?
V2 patch supports reporting node attributes only when updated or re-sync time is elapsed by NM to avoid overhead of the comparing in every heartbeat process. As your advice, logic is similar as reporting node labels. Common logic such as judging whether labels/attributes should be send to RM is extracted as HeartbeatSyncIfNeededHandler, so that handlers of labels and attributes can extend it to reuse the logic. For test cases, I create TestNodeStatusUpdaterForAttributes to test NM side and will add some new test methods in TestResourceTrackerService to test RM side (similar to node labels test cases in TestResourceTrackerService). Thoughts?
> Updating distributed node attributes only when necessary
> --------------------------------------------------------
>
> Key: YARN-8925
> URL: https://issues.apache.org/jira/browse/YARN-8925
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: resourcemanager
> Affects Versions: 3.2.1
> Reporter: Tao Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: performance
> Attachments: YARN-8925.001.patch, YARN-8925.002.patch
>
>
> Currently if distributed node attributes exist, even though there is no change, updating for distributed node attributes will happen in every heartbeat between NM and RM. Updating process will hold NodeAttributesManagerImpl#writeLock and may have some influence in a large cluster. We have found nodes UI of a large cluster is opened slowly and most time it's waiting for the lock in NodeAttributesManagerImpl. I think this updating should be called only when necessary to enhance the performance of related process.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org