You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ramya Sunil (Created) (JIRA)" <ji...@apache.org> on 2011/10/26 01:48:32 UTC
[jira] [Created] (MAPREDUCE-3272) Lost NMs fail to rejoin
Lost NMs fail to rejoin
-----------------------
Key: MAPREDUCE-3272
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Ramya Sunil
Fix For: 0.23.0
Lost nodemanagers fail to join back.
When the NM is lost, RM log reads
{noformat}
INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
{noformat}
When the NM joins back, RM log reads
{noformat}
INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
{noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160172#comment-13160172 ]
Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------
One thing to note. I must wait the full 10 minutes to restart the nodemanager, otherwise I nodemanager is killed with this message.
2011-11-30 11:18:43,336 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved machine.example.com to /default-rack
2011-11-30 11:18:43,336 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Duplicate registration from the node at: machine.example.com, Sending SHUTDOWN Signal to the NodeManager
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles resolved MAPREDUCE-3272.
----------------------------------------
Resolution: Duplicate
After analyzing this issue, I found that it is a dup of MAPREDUCE-3034.
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160173#comment-13160173 ]
Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------
If I use ephemeral ports for the nodemanager, then I do note need to wait for the lost node time out before restarting the nodemanager.
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Ramya Sunil (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160225#comment-13160225 ]
Ramya Sunil commented on MAPREDUCE-3272:
----------------------------------------
Jonathan, the issue which I described was when the nodemanagers are lost and not killed i.e. kill -SIGSTOP 4029 and not kill 4029. This would suspend the nodemanager momentarily but not kill the process. To simulate rejoining of nodemanagers, I did not restart the nodemanager but used kill -SIGCONT 4029 to resume execution.
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jonathan Eagles reassigned MAPREDUCE-3272:
------------------------------------------
Assignee: Jonathan Eagles
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160166#comment-13160166 ]
Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------
Ramya, I could use your help trying to reproduce this.
Here is my setup.
* Not using ephemeral ports for NM
* Single node cluster setup
* Node manager time out is set to 10 minutes
# First the cluster is up and running
$ jps
1761 ResourceManager
4369 Jps
1641 DataNode
1560 NameNode
4029 NodeManager
1840 JobHistoryServer
11996 Launcher
# Next I kill the node manager
$ kill 4029
# I verify the node manager is killed
$ jps
4385 Jps
1761 ResourceManager
1641 DataNode
1560 NameNode
1840 JobHistoryServer
11996 Launcher
# I wait the full 10 minutes for the node manager to time out
2011-11-30 10:45:26,464 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing machine.example.com:45454 of type EXPIRE
2011-11-30 10:45:26,465 INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:machine.example.com:45454 Timed out after 600 secs
2011-11-30 10:45:26,504 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node machine.example.com:45454
2011-11-30 10:45:26,505 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: machine.example.com:45454 Node Transitioned from RUNNING to LOST
# Total nodes on the RM is decremented by one. Lost nodes on the RM is incremented by one
# Then I restart the nodemanager
$ yarn-daemon.sh start nodemanager
# Verify the nodemanager connects successfully in the logs
2011-11-30 11:13:50,315 INFO org.apache.hadoop.yarn.util.RackResolver: Resolved machine.example.com to /default-rack
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing machine.example.com:45454 of type STARTED
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: machine.example.com:45454 Node Transitioned from NEW to RUNNING
2011-11-30 11:13:50,328 INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: NodeManager from node machine.example.com(cmPort: 45454 httpPort: 9999) registered with capability: 8192, assigned nodeId machine.example.com:45454
# I notice the total nodes on the RM is incremented by one. Lost nodes is NOT decremented by one MAPREDUCE-3271.
# Node manager is again available for use. Expected behavior.
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Jonathan Eagles (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13160243#comment-13160243 ]
Jonathan Eagles commented on MAPREDUCE-3272:
--------------------------------------------
Thanks, Ramya for the quick reply. I am now able to reproduce this issue. I have set the yarn.resourcemanager.nm.liveness-monitor.expiry-interval-ms property much lower this time to save myself some grief :)
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Assignee: Jonathan Eagles
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3272) Lost NMs fail to rejoin
Posted by "Hitesh Shah (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3272?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Hitesh Shah updated MAPREDUCE-3272:
-----------------------------------
Priority: Critical (was: Major)
> Lost NMs fail to rejoin
> -----------------------
>
> Key: MAPREDUCE-3272
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3272
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Ramya Sunil
> Priority: Critical
> Fix For: 0.23.1
>
>
> Lost nodemanagers fail to join back.
> When the NM is lost, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.util.AbstractLivelinessMonitor: Expired:<host:port> Timed out after 600 secs
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Processing <host:port> of type EXPIRE
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: Removed Node <host:port>
> INFO org.apache.hadoop.yarn.server.resourcemanager.rmnode.RMNodeImpl: <host:port> Node Transitioned from RUNNING to LOST
> {noformat}
> When the NM joins back, RM log reads
> {noformat}
> INFO org.apache.hadoop.yarn.server.resourcemanager.ResourceTrackerService: Node not found rebooting <host:port>
> {noformat}
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira