You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Bibin Chundatt (Jira)" <ji...@apache.org> on 2020/07/20 06:28:00 UTC
[jira] [Comment Edited] (YARN-10352) Skip schedule on not
heartbeated nodes in Multi Node Placement
[ https://issues.apache.org/jira/browse/YARN-10352?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17160936#comment-17160936 ]
Bibin Chundatt edited comment on YARN-10352 at 7/20/20, 6:27 AM:
-----------------------------------------------------------------
[~prabhujoseph]
With current approach we are iterating through all the nodes 2 times in the partition.
We could filter out the nodes during the {{reSortClusterNodes}} iteration than creating a list then iterating it all over it again. thoughts ?
One more additional filter to {{preferrednodeIterator}} while querying nodes per schedulerKey would reduce the node selection being done during sorting interval of 5 sec.
Iterators.filter(iterator, <filter predicate>
was (Author: bibinchundatt):
[~prabhujoseph]
With current approach we are iterating through all the nodes 2 times in the partition.
We could filter out the nodes during the {{reSortClusterNodes}} iteration that creating a list then iterating it all over it again. thoughts ?
One more additional filter to {{preferrednodeIterator}} while querying nodes per schedulerKey would reduce the node selection being done during sorting interval of 5 sec.
Iterators.filter(iterator, <filter predicate>
> Skip schedule on not heartbeated nodes in Multi Node Placement
> --------------------------------------------------------------
>
> Key: YARN-10352
> URL: https://issues.apache.org/jira/browse/YARN-10352
> Project: Hadoop YARN
> Issue Type: Bug
> Affects Versions: 3.3.0, 3.4.0
> Reporter: Prabhu Joseph
> Assignee: Prabhu Joseph
> Priority: Major
> Labels: capacityscheduler, multi-node-placement
> Attachments: YARN-10352-001.patch
>
>
> When Node Recovery is Enabled, Stopping a NM won't unregister to RM. So RM Active Nodes will be still having those stopped nodes until NM Liveliness Monitor Expires after configured timeout (yarn.nm.liveness-monitor.expiry-interval-ms = 10 mins). During this 10mins, Multi Node Placement assigns the containers on those nodes. They need to exclude the nodes which has not heartbeated for configured heartbeat interval (yarn.resourcemanager.nodemanagers.heartbeat-interval-ms=1000ms) similar to Asynchronous Capacity Scheduler Threads. (CapacityScheduler#shouldSkipNodeSchedule)
> *Repro:*
> 1. Enable Multi Node Placement (yarn.scheduler.capacity.multi-node-placement-enabled) + Node Recovery Enabled (yarn.node.recovery.enabled)
> 2. Have only one NM running say worker0
> 3. Stop worker0 and start any other NM say worker1
> 4. Submit a sleep job. The containers will timeout as assigned to stopped NM worker0.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org