You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Mahadev konar (JIRA)" <ji...@apache.org> on 2011/09/09 01:18:09 UTC
[jira] [Created] (MAPREDUCE-2961) Increase the default threadpool
fize for container launching in the application master.
Increase the default threadpool fize for container launching in the application master.
---------------------------------------------------------------------------------------
Key: MAPREDUCE-2961
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
Project: Hadoop Map/Reduce
Issue Type: Improvement
Components: mrv2
Affects Versions: 0.23.0
Reporter: Mahadev konar
Assignee: Mahadev konar
Fix For: 0.23.0
Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Ted Yu (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ted Yu updated MAPREDUCE-2961:
------------------------------
Summary: Increase the default threadpool size for container launching in the application master. (was: Increase the default threadpool fize for container launching in the application master.)
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2961:
-------------------------------------
Status: Open (was: Patch Available)
Mahadev, you need to rebase after MAPREDUCE-2864. Thanks.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114227#comment-13114227 ]
Hudson commented on MAPREDUCE-2961:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #20 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/20/])
Merge -r 1175293:1175294 from trunk to branch-0.23 to fix MAPREDUCE-2961.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175295
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113209#comment-13113209 ]
Mahadev konar commented on MAPREDUCE-2961:
------------------------------------------
nice patch, looks good to me! Will commit after hudson +1's.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2961:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Vinod!
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112938#comment-13112938 ]
Hadoop QA commented on MAPREDUCE-2961:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496181/MAPREDUCE-2961.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/831//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/831//console
This message is automatically generated.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Attachment: MAPREDUCE-2961-20110923.txt
Attaching patch that should fix this.
bq. The way it is, the pool size grows to the configured core-pool-size and stays there. We should set core-pool-size to be very low, set a max-pool-size and bound the queue size.
We don't need to do this anymore, as the corepool itself is changed according to container-demand.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Status: Patch Available (was: Open)
For very large clusters, one can up the limit(default 500) to increase scalability via an expert-level, non-documented configuration.
The default stack-size for a thread is 500K, so given 2GB default vmem for AM, default upper limit of 500 should be fine.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100803#comment-13100803 ]
Hadoop QA commented on MAPREDUCE-2961:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493691/MAPREDUCE-2961.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/665//console
This message is automatically generated.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114143#comment-13114143 ]
Hadoop QA commented on MAPREDUCE-2961:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496379/MAPREDUCE-2961.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/851//console
This message is automatically generated.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13112914#comment-13112914 ]
Mahadev konar commented on MAPREDUCE-2961:
------------------------------------------
@Vinod,
I do agree this is a hack but unless we see that it doesnt work, I probably think we should not spend too much time on estimating the right amount of threads. Thoughts?
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
fize for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2961:
-------------------------------------
Attachment: MAPREDUCE-2961.patch
Trivial patch to increase the number of threads in the launcher so that we do not bottleneck on container launching.
> Increase the default threadpool fize for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Status: Open (was: Patch Available)
Forgot about this one.
It is easy to set the number of threads dynamically. We just need to set it equal to the number of nodes in the system. We are looking at a scalability of 10,000 nodes, so I'd put a cap of 2000 threads, hardcoded.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13114238#comment-13114238 ]
Hudson commented on MAPREDUCE-2961:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #25 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/25/])
Merge -r 1175293:1175294 from trunk to branch-0.23 to fix MAPREDUCE-2961.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1175295
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/main/java/org/apache/hadoop/mapreduce/v2/app/launcher/ContainerLauncherImpl.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113215#comment-13113215 ]
Mahadev konar commented on MAPREDUCE-2961:
------------------------------------------
Vinod,
Doesnt look like there is cap on the thread pool size. Should we put in cap at 300-400 or so?
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113486#comment-13113486 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2961:
----------------------------------------------------
Oh, BTW, I took Karam's help to see this patch's effect. Before this patch, we had 200 core-pool size for a 100,000 map-only jobs on ~350 nodes. With this patch, which automatically takes care of the thread-pool size(and sets to ~350 for this job in particular), we saw ~30% improvement on wall-clock time.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2961:
-------------------------------------
Attachment: MAPREDUCE-2961.patch
the same patch on top of new changes.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Attachment: MAPREDUCE-2961-20110923.1.txt
Updating patch, including the limit on the number of threads.
With this patch, in most cases, the thread-pool should scale according to the need.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2961:
-------------------------------------
Priority: Blocker (was: Major)
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli reassigned MAPREDUCE-2961:
--------------------------------------------------
Assignee: Vinod Kumar Vavilapalli (was: Mahadev konar)
There are a couple of other bugs that need to be fixed related to ContainerLauncher's thread pool.
- The way it is, the pool size grows to the configured core-pool-size and stays there. We should set core-pool-size to be very low, set a max-pool-size and bound the queue size.
- We also pre-allocate threads which affects MRAppMaster initialization time.
I am taking this up.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113219#comment-13113219 ]
Hadoop QA commented on MAPREDUCE-2961:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496232/MAPREDUCE-2961-20110923.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/835//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/835//console
This message is automatically generated.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Status: Patch Available (was: Open)
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113492#comment-13113492 ]
Hadoop QA commented on MAPREDUCE-2961:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12496276/MAPREDUCE-2961-20110923.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/839//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/839//console
This message is automatically generated.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100996#comment-13100996 ]
Arun C Murthy commented on MAPREDUCE-2961:
------------------------------------------
Vinod, what heuristic do you have in mind?
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13100963#comment-13100963 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-2961:
----------------------------------------------------
I definitely think that instead of a config, this should be a function of #nodes and #tasks with defaults to 10. Will provide a patch for the same.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2961:
-----------------------------------------------
Status: Open (was: Patch Available)
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2961:
-------------------------------------
Attachment: MAPREDUCE-2961.patch
Minor change - move default value to MRJobConfig.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.1.txt, MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
fize for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2961:
-------------------------------------
Status: Patch Available (was: Open)
> Increase the default threadpool fize for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13113217#comment-13113217 ]
Arun C Murthy commented on MAPREDUCE-2961:
------------------------------------------
Anything above 100 threads in a overkill I think... plus each thread has a stack of 1MB. So, we shouldn't be using too many.
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Vinod Kumar Vavilapalli
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961-20110923.txt, MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2961) Increase the default threadpool
size for container launching in the application master.
Posted by "Mahadev konar (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2961?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2961:
-------------------------------------
Status: Patch Available (was: Open)
> Increase the default threadpool size for container launching in the application master.
> ---------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2961
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2961
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Mahadev konar
> Assignee: Mahadev konar
> Priority: Blocker
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2961.patch, MAPREDUCE-2961.patch
>
>
> Currently the default threadpool size is 10 for launching containers in ContainerLauncherImpl. We should increase that to 100 for a reasonable default, so that container launching is not backed up by a small thread pool size.
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira