You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (Created) (JIRA)" <ji...@apache.org> on 2012/03/23 22:37:25 UTC
[jira] [Created] (MAPREDUCE-4061) RM only has 1 AM launcher thread
RM only has 1 AM launcher thread
--------------------------------
Key: MAPREDUCE-4061
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.2
Reporter: Thomas Graves
Priority: Critical
The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239474#comment-13239474 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1032 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1032/])
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305607
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4061) RM only has 1 AM launcher thread
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-4061:
-------------------------------------
Status: Patch Available (was: Open)
I don't see any easy way to unit test this so no test attached.
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238981#comment-13238981 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #721 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/721/])
svn merge -c 1305607 trunk to branch-0.23 FIXES: MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305608)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305608
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Ravi Prakash (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238705#comment-13238705 ]
Ravi Prakash commented on MAPREDUCE-4061:
-----------------------------------------
+1 lgtm
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238851#comment-13238851 ]
Robert Joseph Evans commented on MAPREDUCE-4061:
------------------------------------------------
The patch looks simple enough, and looks like it does what we want. +1
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13239427#comment-13239427 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #997 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/997/])
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305607
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238930#comment-13238930 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #740 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/740/])
svn merge -c 1305607 trunk to branch-0.23 FIXES: MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305608)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305608
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238760#comment-13238760 ]
Hadoop QA commented on MAPREDUCE-4061:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12519993/MAPREDUCE-4061.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2104//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2104//console
This message is automatically generated.
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238953#comment-13238953 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #731 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/731/])
svn merge -c 1305607 trunk to branch-0.23 FIXES: MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305608)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305608
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-4061) RM only has 1 AM launcher thread
Posted by "Thomas Graves (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves reassigned MAPREDUCE-4061:
----------------------------------------
Assignee: Thomas Graves
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4061) RM only has 1 AM launcher thread
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-4061:
-------------------------------------
Priority: Blocker (was: Critical)
Target Version/s: 0.23.2 (was: 0.23.3)
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238961#comment-13238961 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1932 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1932/])
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305607
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238986#comment-13238986 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2007 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2007/])
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305607
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4061) RM only has 1 AM launcher thread
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-4061:
-------------------------------------
Attachment: MAPREDUCE-4061.patch
increased the number of threads in AM launch to 10 and then also saw the same bug in the MR app master taskCleaner, so increased that to 5.
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4061) RM only has 1 AM launcher
thread
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13238929#comment-13238929 ]
Hudson commented on MAPREDUCE-4061:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1943 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1943/])
MAPREDUCE-4061. RM only has 1 AM launcher thread (tgraves via bobby) (Revision 1305607)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1305607
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/taskclean/TaskCleanerImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/amlauncher/ApplicationMasterLauncher.java
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4061) RM only has 1 AM launcher thread
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4061?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4061:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 0.23.2
Status: Resolved (was: Patch Available)
Thnaks Tom.
I just put this into trunk, branch-0.23 and branch-0.23.2
> RM only has 1 AM launcher thread
> --------------------------------
>
> Key: MAPREDUCE-4061
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4061
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.2
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Blocker
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-4061.patch
>
>
> The application master launcher has a thread pool that is configured with core size 1, maximum 10. The thread pool will not create over
> the core size thread unless the queue it is using is full. We are using an unbounded queue, so the thread pool will only ever create 1 thread. We need to have more then 1 AM launch thread.
> If that thread becomes hung for some reason, the RM can no longer launch any application masters. We have seen an instance of this when a NM become unresponsive - something bad happened to host, not sure what yet.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira