You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Jason Lowe (Created) (JIRA)" <ji...@apache.org> on 2012/01/26 01:12:38 UTC
[jira] [Created] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Allow restarted NM to rejoin cluster before RM expires it
---------------------------------------------------------
Key: MAPREDUCE-3730
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2, resourcemanager
Affects Versions: 0.23.1, 0.24.0
Reporter: Jason Lowe
Assignee: Jason Lowe
When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3730:
-----------------------------------------------
Priority: Minor (was: Major)
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Also given the default setting for the nodes is ephemeral ports, I think this issue is neither major nor critical. I am downgrading it. Let's take the patch to completion irrespective of the priority as you are already close to finish.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3730:
-----------------------------------------------
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Open (was: Patch Available)
This patch has gone a little stale now. Can you please update?
Also, instead of using a new RMNodeImpl, you can simply send a RECONNECTED event to the existing RMNodeImpl. I think we should also not process the health status during registration. That will happen anyways in the next status update, right?
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193871#comment-13193871 ]
Jason Lowe commented on MAPREDUCE-3730:
---------------------------------------
The test failures are unrelated to the patch. See MAPREDUCE-3729.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216424#comment-13216424 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #966 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/966/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe. (Revision 1293436)
Result = FAILURE
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215968#comment-13215968 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #589 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/589/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe.
svn merge --ignore-ancestry -c 1293436 ../../trunk (Revision 1293439)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293439
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215959#comment-13215959 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1848 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1848/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe. (Revision 1293436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216038#comment-13216038 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1785 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1785/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe. (Revision 1293436)
Result = ABORTED
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Target Version/s: 0.23.1, 0.24.0
Status: Patch Available (was: Open)
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Target Version/s: 0.24.0, 0.23.2 (was: 0.24.0, 0.23.1)
Status: Patch Available (was: Open)
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193485#comment-13193485 ]
Hadoop QA commented on MAPREDUCE-3730:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511913/MAPREDUCE-3730.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1679//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1679//console
This message is automatically generated.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3730:
-----------------------------------------------
Resolution: Fixed
Target Version/s: 0.23.2 (was: 0.23.2, 0.24.0)
Release Note: Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Just committed this to trunk and branch-0.23. Thanks Jason!
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216036#comment-13216036 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #591 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/591/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe.
svn merge --ignore-ancestry -c 1293436 ../../trunk (Revision 1293439)
Result = ABORTED
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293439
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215975#comment-13215975 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1774 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1774/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe. (Revision 1293436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Attachment: MAPREDUCE-3730.patch
Updated patch to merge recent changes on trunk.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216443#comment-13216443 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #207 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/207/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe.
svn merge --ignore-ancestry -c 1293436 ../../trunk (Revision 1293439)
Result = FAILURE
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293439
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13209113#comment-13209113 ]
Jason Lowe commented on MAPREDUCE-3730:
---------------------------------------
Thanks Vinod for taking another look. The race condition mentioned above isn't specific to the new node approach, as it's still present when modifying the original RMNode. A node could rejoin just as the tracker service expires it, and we could mark the node as lost (new or original node object, doesn't matter) just as it starts to heartbeat in again.
I still think using a new node object is easier to implement, understand, and maintain since it more directly mirrors the existing behavior of a node that eventually expires and starts again. However I'll defer to your judgement and update the patch accordingly.
Note that using ephemeral ports can be problematic, as it opens the door to inadvertent duplicate NM registration and the problems described in MAPREDUCE-3363. I believe all of our clusters are currently configured to not use them. We should reconsider the ephemeral port default once this goes in, since it was only changed to workaround the NM rejoin issue.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216458#comment-13216458 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1001 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1001/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe. (Revision 1293436)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293436
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3730:
-----------------------------------------------
Fix Version/s: 0.23.2
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Open (was: Patch Available)
Running through the review queue, arrived at this finally.
First thing, the patch has gone stale again, sorry for the review delay.
Second, I still maintain that you should send the RECONNECTED event to a single (old) RMNode. All the conditional logic depending on whether the old node was healthy or not, is simply out of place.
Even the race condition that you outlined above clearly hints to me that we should have a single node. If we have a single node, we will be able to handle the RECONNECTED event at a LOST state correctly by letting the new node in.
If the RECONNECT event goes to the same node, sure, you have to handle the event in all states, but it makes it that much easier to reason there instead of an unrelated place like ResourceTrackerService.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Attachment: MAPREDUCE-3730.patch
Updated patch to trunk and addressed review comments.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13194902#comment-13194902 ]
Jason Lowe commented on MAPREDUCE-3730:
---------------------------------------
bq. Also, instead of using a new RMNodeImpl, you can simply send a RECONNECTED event to the existing RMNodeImpl.
There are a few reasons I thought using a new RMNodeImpl would be better:
1) The new node could have different capabilities than the previous node (e.g.: more RAM).
2) Reusing the node means all states have to handle the RECONNECTED event. It seemed simpler from a maintenance standpoint to treat the RECONNECTED event as an accelerated EXPIRE/STARTED transition since that's effectively what occurred. The RM just didn't get a chance to notice the EXPIRE event because the timeout was too large in that instance.
bq. I think we should also not process the health status during registration. That will happen anyways in the next status update, right?
Following the "accelerated EXPIRE/STARTED" logic, it seemed better to assume a reconnected node is just like a newly connected node (i.e.: we assume healthy until proven otherwise). As you say, any incorrect guess there will be corrected on the next status update.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13210653#comment-13210653 ]
Hadoop QA commented on MAPREDUCE-3730:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12515032/MAPREDUCE-3730.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1889//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1889//console
This message is automatically generated.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Attachment: MAPREDUCE-3730.patch
Adding a patch that is a variation of the initial patch in MAPREDUCE-3070.
There is one race condition I can think of that isn't addressed, and that's the case where the NM registers with the RM just as the node expiration occurs. If the expiration is processed after the reconnect then the node will still be marked lost. When the node heartbeats back in, the RM will direct it to reboot, and the NM will simply shutdown.
Once MAPREDUCE-3034 is addressed, the NM will restart from the reboot directive and the node will recover.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13216431#comment-13216431 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #179 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/179/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe.
svn merge --ignore-ancestry -c 1293436 ../../trunk (Revision 1293439)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293439
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215950#comment-13215950 ]
Hudson commented on MAPREDUCE-3730:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #576 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/576/])
MAPREDUCE-3730. Modified RM to allow restarted NMs to be able to join the cluster without waiting for expiry. Contributed by Jason Lowe.
svn merge --ignore-ancestry -c 1293436 ../../trunk (Revision 1293439)
Result = SUCCESS
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1293439
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/ResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeEventType.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/rmnode/RMNodeReconnectEvent.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/MockNodes.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestResourceTrackerService.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestCapacityScheduler.java
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13215937#comment-13215937 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3730:
----------------------------------------------------
Thanks Jason, this looks far better. +1.
I found some race conditions in the same code area, will file separate ticket and file them.
I am going to push this in.
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Priority: Minor
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3730) Allow restarted NM to rejoin
cluster before RM expires it
Posted by "Jason Lowe (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3730?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Jason Lowe updated MAPREDUCE-3730:
----------------------------------
Target Version/s: 0.23.1, 0.24.0 (was: 0.24.0, 0.23.1)
Status: Patch Available (was: Open)
> Allow restarted NM to rejoin cluster before RM expires it
> ---------------------------------------------------------
>
> Key: MAPREDUCE-3730
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3730
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.1, 0.24.0
> Reporter: Jason Lowe
> Assignee: Jason Lowe
> Attachments: MAPREDUCE-3730.patch, MAPREDUCE-3730.patch
>
>
> When a node in the RUNNING state (healthy or unhealthy) is rebooted, the resourcemanager rejects the nodemanager's registration request as a duplicate because it is convinced that the nodemanager is already running on that node. It won't allow that node to rejoin the cluster until the node expiration time elapses which is 10min+ by default. We should allow the NM to rejoin the cluster if it re-registers within the expiration timeout.
> Note that this problem occurs with NMs that are configured to specific ports. If ephemeral ports are used then a NM reboot "works" because the RM thinks the NM registration is for a new node. See the discussions in MAPREDUCE-3070 and MAPREDUCE-3363.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira