[jira] [Created] (YARN-2299) inconsistency at identifying node

["Hong Zhiguo (JIRA)" <ji...@apache.org>]

Hong Zhiguo created YARN-2299:
---------------------------------

             Summary: inconsistency at identifying node
                 Key: YARN-2299
                 URL: https://issues.apache.org/jira/browse/YARN-2299
             Project: Hadoop YARN
          Issue Type: Bug
          Components: resourcemanager
            Reporter: Hong Zhiguo
            Assignee: Hong Zhiguo
            Priority: Critical


If port of "yarn.nodemanager.address" is not specified at NM, NM will choose random port. If the NM is ungracefully dead(OOM kill, kill -9, or OS restart) and then restarted within "yarn.nm.liveness-monitor.expiry-interval-ms", "host:port1" and "host:port2" will both be present in "Active Nodes" on WebUI for a while, and after host:port1 expiration, we get host:port1 in "Lost Nodes" and host:port2 in "Active Nodes". If the NM is ungracefully dead again, we get only host:port1 in "Lost Nodes". "host:port2" is neither in "Active Nodes" nor in  "Lost Nodes".

Another case, two NM is running on same host(miniYarnCluster or other test purpose), if both of them are lost, we get only one "Lost Nodes" in WebUI.

In both case, sum of "Active Nodes" and "Lost Nodes" is not the number of nodes we expected.

The root cause is due to inconsistency at how we think two Nodes are identical.
When we manager active nodes(RMContextImpl.nodes), we use NodeId which contains port. Two nodes with same host but different port are thought to be different node.
But when we manager inactive nodes(RMContextImpl.inactiveNodes), we use only use host. Two nodes with same host but different port are thought to identical.

We should differentiate 2 cases: 
 - intentionally multiple NMs per host
 - NM instances one after another on same host

Two possible solutions:
1) Introduce a boolean config like "one-node-per-host"(default as "true"), and use host to differentiate nodes on RM if it's true.

2) Make it mandatory to have valid port in "yarn.nodemanager.address" config.  In this sutiation, NM instances one after another on same host will have same NodeId, while intentionally multiple NMs per host will have different NodeId.

Personally I prefer option 1 because it's easier for users.




--
This message was sent by Atlassian JIRA
(v6.2#6252)