You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Varun Vasudev (JIRA)" <ji...@apache.org> on 2015/11/11 11:58:10 UTC
[jira] [Commented] (YARN-4344) NMs reconnecting with changed
capabilities can lead to wrong cluster resource calculations
[ https://issues.apache.org/jira/browse/YARN-4344?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15000229#comment-15000229 ]
Varun Vasudev commented on YARN-4344:
-------------------------------------
An example of a situation is shown below -
{code}
2015-11-09 10:43:51,784 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node 10.0.0.64(cmPort: 30050 httpPort: 30060) registered with capability: <memory:5632, vCores:8>, assigned nodeId 10.0.0.64:30050
2015-11-09 10:43:51,786 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(434)) - 10.0.0.64:30050 Node Transitioned from NEW to RUNNING
2015-11-09 10:43:51,814 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.64:30050 clusterResource: <memory:5632, vCores:8>
2015-11-09 10:44:37,878 INFO util.RackResolver (RackResolver.java:coreResolve(109)) - Resolved 10.0.0.63 to /default-rack
2015-11-09 10:44:37,879 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node 10.0.0.63(cmPort: 30050 httpPort: 30060) registered with capability: <memory:10240, vCores:4>, assigned nodeId 10.0.0.63:30050
2015-11-09 10:44:37,879 INFO rmnode.RMNodeImpl (RMNodeImpl.java:handle(434)) - 10.0.0.63:30050 Node Transitioned from NEW to RUNNING
2015-11-09 10:44:37,882 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.63:30050 clusterResource: <memory:15872, vCores:12>
2015-11-09 10:44:39,307 INFO util.RackResolver (RackResolver.java:coreResolve(109)) - Resolved 10.0.0.64 to /default-rack
2015-11-09 10:44:39,309 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(313)) - Reconnect from the node at: 10.0.0.64
2015-11-09 10:44:39,312 INFO resourcemanager.ResourceTrackerService (ResourceTrackerService.java:registerNodeManager(345)) - NodeManager from node 10.0.0.64(cmPort: 30050 httpPort: 30060) registered with capability: <memory:10240, vCores:4>, assigned nodeId 10.0.0.64:30050
2015-11-09 10:44:39,314 INFO capacity.CapacityScheduler (CapacityScheduler.java:removeNode(1247)) - Removed node 10.0.0.64:30050 clusterResource: <memory:5632, vCores:8>
2015-11-09 10:44:39,315 INFO capacity.CapacityScheduler (CapacityScheduler.java:addNode(1193)) - Added node 10.0.0.64:30050 clusterResource: <memory:15872, vCores:12>
{code}
In this case - NM's from 10.0.0.64 and 10.0.0.63 registered leading to a total cluster resource of clusterResource: <memory:15872, vCores:12>. After that 10.0.0.64 re-connected with changed capabilities(from <memory:5632, vCores:8> to <memory:10240, vCores:4>). This should have led to the cluster resources becoming <memory:20480, vcores:8> but instead it is calculated to be <memory:15872, vCores:12>.
The root cause is this piece of code from RMNodeImpl -
{code}
rmNode.context.getDispatcher().getEventHandler().handle(
new NodeRemovedSchedulerEvent(rmNode));
if (!rmNode.getTotalCapability().equals(
newNode.getTotalCapability())) {
rmNode.totalCapability = newNode.getTotalCapability();
{code}
If the dispatcher is delayed in its processing of the event, by the time the remove node is processed, rmNode.totalCapability = newNode.getTotalCapability() has already been executed and the resources that are removed are the changed capabilities and not the older capabilities of the node.
> NMs reconnecting with changed capabilities can lead to wrong cluster resource calculations
> ------------------------------------------------------------------------------------------
>
> Key: YARN-4344
> URL: https://issues.apache.org/jira/browse/YARN-4344
> Project: Hadoop YARN
> Issue Type: Bug
> Components: resourcemanager
> Affects Versions: 2.7.1, 2.6.2
> Reporter: Varun Vasudev
> Assignee: Varun Vasudev
> Priority: Critical
>
> After YARN-3802, if an NM re-connects to the RM with changed capabilities, there can arise situations where the overall cluster resource calculation for the cluster will be incorrect leading to inconsistencies in scheduling.
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)