[jira] [Commented] (REEF-627) EvaluatorManagerFactory is not setting the resource memory of the NodeDescriptor correctly

["Andrew Chung (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/REEF-627?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14711635#comment-14711635 ] 

Andrew Chung commented on REEF-627:
-----------------------------------

Is {{YarnContainerManager.onNodesUpdated()}} called at the start of the application to provide all nodes to the AM? I'm failing to find the right method call to retrieve node information from the AM in the Hadoop JavaDocs. If {{YarnContainerManager.onNodesUpdated()}} is called, I suspect we should ever need to handle the case where we need to "fake" the resources here, unless there is a bug in Hadoop. In any case, it might be better to provide an invalid mark instead of using the Container's Resource for the entire node/rack, and update the invalid mark when {{onNodesUpdated()}} is called with the correct values. How does this sound?

> EvaluatorManagerFactory is not setting the resource memory of the NodeDescriptor correctly
> ------------------------------------------------------------------------------------------
>
>                 Key: REEF-627
>                 URL: https://issues.apache.org/jira/browse/REEF-627
>             Project: REEF
>          Issue Type: Bug
>          Components: REEF Driver
>            Reporter: Andrew Chung
>
> As seen here:
> {code}
> final NodeDescriptorEvent nodeDescriptorEvent = NodeDescriptorEventImpl.newBuilder().setIdentifier(nodeId)
>           .setHostName(hostNameAndPort[0]).setPort(Integer.parseInt(hostNameAndPort[1]))
>           .setMemorySize(resourceAllocationEvent.getResourceMemory())
>           .setRackName(resourceAllocationEvent.getRackName().get()).build();
> {code}
> We are setting the amount of node memory to the amount of container memory, which is incorrect.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)