You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Harsh J (Created) (JIRA)" <ji...@apache.org> on 2012/03/16 22:23:41 UTC
[jira] [Created] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Lower minimum-allocation-mb to sensible defaults
------------------------------------------------
Key: MAPREDUCE-4026
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, scheduler
Affects Versions: 0.23.3
Reporter: Harsh J
Assignee: Harsh J
The CapacityScheduler's minimum-allocation-mb is set to 1024.
The FIFO's minimum-allocation-mb meanwhile, is 128.
I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258755#comment-13258755 ]
Hadoop QA commented on MAPREDUCE-4026:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523606/MAPREDUCE-4026.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2266//console
This message is automatically generated.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258185#comment-13258185 ]
Hadoop QA commented on MAPREDUCE-4026:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523483/MAPREDUCE-4026.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2260//console
This message is automatically generated.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Attachment: MAPREDUCE-4026.patch
Bobby - Good point, I should've done that!
Addressed now, and also fixed some references to the MINIMUM_ALLOCATION config where applicable. MAXIMUM_ALLOCATION doesn't seem to be tested/used for at the moment, and I've filed MAPREDUCE-4171 for that.
Thanks for taking a look, please do review again!
To be sure: This only changes minimum allocation values, not job default memory in MR. DistributedShell too in its test uses 128 MB as the min-alloc, for example. So I don't see this change regressing any working features.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Attachment: MAPREDUCE-4026.patch
Didn't realize there was a hardcoded test in TestFifoScheduler. Fixed test to not hardcode now, and rather reuse a constant.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231714#comment-13231714 ]
Arun C Murthy commented on MAPREDUCE-4026:
------------------------------------------
You can't really do much in jvm with 128M heap. Maybe 512M ...
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 0.23.3
> Reporter: Harsh J
> Assignee: Harsh J
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Target Version/s: 2.0.0 (was: trunk, 2.0.0)
Status: Patch Available (was: Open)
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 0.23.3
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Attachment: MAPREDUCE-4026.patch
Only TestRMWebServices failure seems related to this. Fixed that test to not use hardcodes either. Fixed test passes now.
The other failures stem out of port binding issues (mis-closed min-YARN clusters?) and a couple of others with kerberos-related issues.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258351#comment-13258351 ]
Hadoop QA commented on MAPREDUCE-4026:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523508/MAPREDUCE-4026.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2264//console
This message is automatically generated.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Resolution: Duplicate
Status: Resolved (was: Patch Available)
Carrying this same effort on MAPREDUCE-3812 along with other relevant and already existing changes there.
Resolving as duplicate of MAPREDUCE-3812.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Affects Version/s: (was: 0.23.3)
2.0.0
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258284#comment-13258284 ]
Hadoop QA commented on MAPREDUCE-4026:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523495/MAPREDUCE-4026.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2263//console
This message is automatically generated.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Robert Joseph Evans (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258368#comment-13258368 ]
Robert Joseph Evans commented on MAPREDUCE-4026:
------------------------------------------------
I like the patch, I don't really have much of an opinion of the changing of the default values. I only have one NIT and that is with preexisting code. I would like to have MINIMUM_MEMORY and MAXIMUM_MEMORY in both the fifo and the capacity scheduler renamed to DEFAULT_MINIMUM_ALLOCATION and DEFAULT_MAXIMUM_ALLOCATION respectively. This is just so it is less confusing what these values are, especially now that they are exposed to the tests.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Attachment: MAPREDUCE-4026.patch
Patch that lowers the min-alloc MB to 128 in both existing FIFO and CS schedulers.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13231832#comment-13231832 ]
Harsh J commented on MAPREDUCE-4026:
------------------------------------
Arun,
It is only the min-alloc am changing, not the job defaults (mapreduce.map.memory.mb and such) which are presently 1024 or higher, and will need a new JIRA if to be lowered as a default (agree that 128 is too low - either 200 or 512 should be it for those).
I don't think lowering min alloc default would change default job requirements, but will at least allow folks to pass -D options for memory needs that are lower than 1024 - for more tasks in parallel.
Or am I missing something?
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 0.23.3
> Reporter: Harsh J
> Assignee: Harsh J
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-4026:
-------------------------------------------
Target Version/s: 2.0.0, trunk (was: 0.23.3)
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 0.23.3
> Reporter: Harsh J
> Assignee: Harsh J
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258218#comment-13258218 ]
Hadoop QA commented on MAPREDUCE-4026:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523491/MAPREDUCE-4026.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2262//console
This message is automatically generated.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4026) Lower minimum-allocation-mb to
sensible defaults
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4026?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4026:
-------------------------------
Attachment: MAPREDUCE-4026.patch
Ah earlier patch lacked all sub commits. Re-upping a new one.
> Lower minimum-allocation-mb to sensible defaults
> ------------------------------------------------
>
> Key: MAPREDUCE-4026
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4026
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, scheduler
> Affects Versions: 2.0.0
> Reporter: Harsh J
> Assignee: Harsh J
> Attachments: MAPREDUCE-4026.patch, MAPREDUCE-4026.patch, MAPREDUCE-4026.patch
>
>
> The CapacityScheduler's minimum-allocation-mb is set to 1024.
> The FIFO's minimum-allocation-mb meanwhile, is 128.
> I propose changing the formers' minimum to that amount as well. 1024 is way too much as a default, wastes "slots" on NMs - and I also do not see why CS has to deviate that settings from the FIFO default.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira