[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin

["Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160166#comment-13160166 ] 

Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------

Ramya, I could use your help trying to reproduce this.

Here is my setup.
* Not using ephemeral ports for NM
* Single node cluster setup
* Node manager time out is set to 10 minutes

# First the cluster is up and running
$ jps
1761 ResourceManager
4369 Jps
1641 DataNode
1560 NameNode
4029 NodeManager
1840 JobHistoryServer
11996 Launcher

# Next I kill the node manager
$ kill 4029

# I verify the node manager is killed
$ jps
4385 Jps
1761 ResourceManager
1641 DataNode
1560 NameNode
1840 JobHistoryServer
11996 Launcher

# I wait the full 10 minutes for the node manager to time out
2011-11-30 10:45:26,464 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing machine.example.com:45454 of type EXPIRE
2011-11-30 10:45:26,465 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:machine.example.com:45454 Timed out after 600 secs
2011-11-30 10:45:26,504 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node machine.example.com:45454
2011-11-30 10:45:26,505 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: machine.example.com:45454 Node Transitioned from RUNNING to LOST

# Total nodes on the RM is decremented by one. Lost nodes on the RM is incremented by one
# Then I restart the nodemanager
$ yarn-daemon.sh start nodemanager

# Verify the nodemanager connects successfully in the logs
2011-11-30 11:13:50,315 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved machine.example.com to /default-rack
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing machine.example.com:45454 of type STARTED
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: machine.example.com:45454 Node Transitioned from NEW to RUNNING
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node machine.example.com(cmPort: 45454 httpPort: 9999) registered with capability: 8192, assigned nodeId machine.example.com:45454

# I notice the total nodes on the RM is incremented by one. Lost nodes is NOT decremented by one MAPREDUCE-3271.
# Node manager is again available for use. Expected behavior.
                
> Lost NMs fail to rejoin
> -----------------------
>
>                 Key: MAPREDUCE-3272
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Ramya Sunil
>            Assignee: Jonathan Eagles
>            Priority: Critical
>             Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back. 
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira