You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Vinod Kumar Vavilapalli (Created) (JIRA)" <ji...@apache.org> on 2012/02/05 23:43:59 UTC
[jira] [Created] (MAPREDUCE-3812) Change default AM slot size to be
1GB
Change default AM slot size to be 1GB
-------------------------------------
Key: MAPREDUCE-3812
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
Project: Hadoop Map/Reduce
Issue Type: Sub-task
Reporter: Vinod Kumar Vavilapalli
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259000#comment-13259000 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523665/MAPREDUCE-3812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2269//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2269//console
This message is automatically generated.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Harsh J (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 0.23.3
Fix Version/s: (was: 0.23.2)
Setting a target version instead of a fix version.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259668#comment-13259668 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2190 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2190/])
MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Attachment: MAPREDUCE-3812-20120206.txt
Correct patch applicable to trunk.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201995#comment-13201995 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513549/MAPREDUCE-3812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 12 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1807//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1807//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200978#comment-13200978 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513379/MAPREDUCE-3812-20120205.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1782//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Status: Open (was: Patch Available)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Status: Patch Available (was: Open)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Description:
After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
This is just a configuration change.
[P.s.]:
- Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
- The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
- 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
was:
After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
This is just a configuration change.
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Summary: Lower default allocation sizes, fix allocation configurations and document them (was: Change default memory slot sizes to be 1.5GB)
Updated title and description
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259001#comment-13259001 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523665/MAPREDUCE-3812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2270//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2270//console
This message is automatically generated.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259670#comment-13259670 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2116 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2116/])
MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260526#comment-13260526 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1059 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1059/])
MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Status: Open (was: Patch Available)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Status: Patch Available (was: Open)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Attachment: MAPREDUCE-3812-20120206.1.txt
Fixing tests.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Attachment: MAPREDUCE-3812.patch
Off the failures, only TestRMContainerAllocator looked relevant. Fixed that test by removing an unnecessary init override (could've also called super.init but the whole method override was unneeded, I had it carried from the earlier patch).
Here are new test results, from those that failed, per project:
*YARN Server*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.472 sec
Running org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.826 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 4.453 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 5.21 sec
Running org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 1.997 sec
Results :
Tests run: 8, Failures: 0, Errors: 0, Skipped: 0
{code}
*MapReduce Client App*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 51.106 sec
Results :
Tests run: 9, Failures: 0, Errors: 0, Skipped: 0
{code}
*MR JobClient*:
{code}
-------------------------------------------------------
T E S T S
-------------------------------------------------------
-------------------------------------------------------
T E S T S
-------------------------------------------------------
Running org.apache.hadoop.mapred.TestClientRedirect
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 14.422 sec
Running org.apache.hadoop.mapreduce.security.TestJHSSecurity
Tests run: 1, Failures: 0, Errors: 0, Skipped: 0, Time elapsed: 6.899 sec
Results :
Tests run: 2, Failures: 0, Errors: 0, Skipped: 0
{code}
All failed tests have passed locally. On jenkins it looks like some other test failing to shutdown YARN services in teardown methods is causing this issue. Resubmitting.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Hadoop Flags: Incompatible change
Ah and yes this is an incompatible change as it changes the way min/max alloc configs are read. With this patch, no more per-scheduler min/max alloc configs exist.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Fix Version/s: (was: 0.23.1)
0.23.2
Status: Open (was: Patch Available)
This patch needs more work. Moving to 0.23.2.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.2
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Status: Open (was: Patch Available)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Release Note:
Removed two sets of previously available config properties:
1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )
In favor of two new, generically named properties:
1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.
Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.
Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260500#comment-13260500 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1024 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1024/])
MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Attachment: MAPREDUCE-3812.patch
Fixed the test.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258938#comment-13258938 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12523646/MAPREDUCE-3812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 5 new or modified test files.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.TestContainerManagerSecurity
org.apache.hadoop.yarn.server.resourcemanager.security.TestApplicationTokens
org.apache.hadoop.yarn.server.resourcemanager.TestClientRMService
org.apache.hadoop.yarn.server.resourcemanager.resourcetracker.TestNMExpiry
org.apache.hadoop.yarn.server.resourcemanager.TestAMAuthorization
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationACLs
org.apache.hadoop.mapreduce.v2.app.TestRMContainerAllocator
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapreduce.security.TestJHSSecurity
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2268//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2268//console
This message is automatically generated.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 2.0.0, trunk (was: 0.23.3)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259660#comment-13259660 ]
Robert Joseph Evans commented on MAPREDUCE-3812:
------------------------------------------------
The changes look OK to me. +1. I am not really sure about the need to make the different configurations common, but looking at the patch for the fair scheduler it uses the same concept so I am OK with the change.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default AM slot size to be
1GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Description:
After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
This is just a configuration change.
Affects Version/s: 0.23.0
Fix Version/s: 0.23.1
Assignee: Vinod Kumar Vavilapalli
> Change default AM slot size to be 1GB
> -------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Status: Patch Available (was: Open)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Attachment: MAPREDUCE-3812.patch
Fixed unit tests.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201534#comment-13201534 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513476/MAPREDUCE-3812-20120206.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.TestApplicationCleanup
org.apache.hadoop.yarn.server.resourcemanager.TestFifoScheduler
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServices
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1791//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Status: Open (was: Patch Available)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259003#comment-13259003 ]
Harsh J commented on MAPREDUCE-3812:
------------------------------------
Failing test is now unrelated to patch. Please review!
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13260477#comment-13260477 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #237 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/237/])
svn merge -c 1329282 FIXES: MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329285)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329285
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Summary: Change default memory slot sizes to be 1.5GB (was: Change default AM slot size to be 1GB)
Digging more, I realized we've been testing with 1.5 slot size for normal tasks and the same 1.5GB for AM since a long time. I am going to change the defaults to reflect what we've been testing with.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201706#comment-13201706 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513504/MAPREDUCE-3812-20120206.1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed the unit tests build
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1799//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1799//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3812:
-------------------------------------
Status: Patch Available (was: Open)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Attachment: MAPREDUCE-3812-20120205.txt
- Changed default minimum-allocation-mb to be 1536, maximum-allocation-mb to be 6*1536, default total memory per node to be 6*1536.
- Changed default slot size for AM to be 1536, heap size to be 1024mb.
- Also removed the per-scheduler min and max allocation mb configurations. These are common concepts across schedulers.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Attachment: MAPREDUCE-3812.patch
Here is a fresh rewrite patch as the last one available here was out of sync and I figured its faster rewriting changes and gathering MAPREDUCE-4026 and MAPREDUCE-4027 rather than trying to resolve the out-of-sync conflicts.
I've not touched some tests as I felt they were unnecessary. I ran 4 tests for this change: TestFifoScheduler, TestRMWebServices, TestRM, TestApplicationCleanup and all of them pass.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13259678#comment-13259678 ]
Hudson commented on MAPREDUCE-3812:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2132 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2132/])
MAPREDUCE-3812. Lower default allocation sizes, fix allocation configurations and document them (Harsh J via bobby) (Revision 1329282)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1329282
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-app/src/test/java/org/apache/hadoop/mapreduce/v2/app/TestRMContainerAllocator.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/java/org/apache/hadoop/mapreduce/MRJobConfig.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-core/src/main/resources/mapred-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-mapreduce-client/hadoop-mapreduce-client-hs/src/test/resources/job_1329348432655_0001_conf.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-applications/hadoop-yarn-applications-distributedshell/src/test/java/org/apache/hadoop/yarn/applications/distributedshell/TestDistributedShell.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/java/org/apache/hadoop/yarn/conf/YarnConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common/src/main/resources/yarn-default.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/TestFifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServices.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/ClusterSetup.apt.vm
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13258899#comment-13258899 ]
Harsh J commented on MAPREDUCE-3812:
------------------------------------
Didn't realize this patch changed the way allocs would get read, or that it was doing more than just changing default AM memory requirements. Would save me a lot of work I've otherwise been doing on MAPREDUCE-4026 and MAPREDUCE-4027.
I'll take over this one since it looks to have gone stale. Thanks for revamping the whole default way of reading the allocs - much simpler way now. No idea why there was per-scheduler configs before when you can't use more than one scheduler anyway.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13202089#comment-13202089 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513566/MAPREDUCE-3812.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 15 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestMiniMRBringup
org.apache.hadoop.mapred.TestMiniMRChildTask
org.apache.hadoop.mapred.TestReduceFetch
org.apache.hadoop.mapred.TestReduceFetchFromPartialMem
org.apache.hadoop.mapred.TestJobCounters
org.apache.hadoop.mapreduce.TestChild
org.apache.hadoop.mapred.TestMiniMRClientCluster
org.apache.hadoop.mapreduce.v2.TestSpeculativeExecution
org.apache.hadoop.mapreduce.lib.output.TestJobOutputCommitter
org.apache.hadoop.mapred.TestClientRedirect
org.apache.hadoop.mapred.TestLazyOutput
org.apache.hadoop.mapreduce.TestMapReduceLazyOutput
org.apache.hadoop.mapred.TestSpecialCharactersInOutputPath
org.apache.hadoop.mapreduce.v2.TestRMNMInfo
org.apache.hadoop.mapred.TestJobSysDirWithDFS
org.apache.hadoop.mapreduce.v2.TestMiniMRProxyUser
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1809//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1809//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Attachment: MAPREDUCE-3812.patch
One micro change: Updating mapred-default's AM memory value as a new default entry was added via MAPREDUCE-3947 recently.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3812:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 3.0.0
2.0.0
0.23.3
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Status: Resolved (was: Patch Available)
Thanks Harsh,
I put this into branch-0.23, branch-2, and trunk
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J reassigned MAPREDUCE-3812:
----------------------------------
Assignee: Harsh J (was: Vinod Kumar Vavilapalli)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Release Note:
Removes two sets of previously available config properties:
1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )
In favor of two new, generically named properties:
1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.
Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.
Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
was:
Removed two sets of previously available config properties:
1. ( yarn.scheduler.fifo.minimum-allocation-mb and yarn.scheduler.fifo.maximum-allocation-mb ) and,
2. ( yarn.scheduler.capacity.minimum-allocation-mb and yarn.scheduler.capacity.maximum-allocation-mb )
In favor of two new, generically named properties:
1. yarn.scheduler.minimum-allocation-mb - This acts as the floor value of memory resource requests for containers.
2. yarn.scheduler.maximum-allocation-mb - This acts as the ceiling value of memory resource requests for containers.
Both these properties need to be set at the ResourceManager (RM) to take effect, as the RM is where the scheduler resides.
Also changes the default minimum and maximums to 128 MB and 10 GB respectively.
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Fix For: 0.23.3, 2.0.0, 3.0.0
>
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Lower default allocation sizes,
fix allocation configurations and document them
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-3812:
-------------------------------
Target Version/s: 2.0.0, trunk (was: trunk, 2.0.0)
Status: Patch Available (was: Open)
> Lower default allocation sizes, fix allocation configurations and document them
> -------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Harsh J
> Attachments: MAPREDUCE-3812-20120205.txt, MAPREDUCE-3812-20120206.1.txt, MAPREDUCE-3812-20120206.txt, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch, MAPREDUCE-3812.patch
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
> [P.s.]:
> - Currently min/max alloc is set at a per-scheduler config level, which makes no sense as there's no way to run multiple schedulers anyway. Switch configs to use a generic RM-config.
> - The min/max alloc configs aren't documented and we ought to document it (i.e. MAPREDUCE-4027)
> - 1 GB is perhaps too high for a slot's minimum. While job defaults can be left at such values, we should lower the minimum alloc to 128 MB to allow special requests of low memory out of the box itself. Shouldn't impact MR App in any way.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3812) Change default memory slot sizes
to be 1.5GB
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3812:
-----------------------------------------------
Status: Patch Available (was: Open)
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3812) Change default memory slot
sizes to be 1.5GB
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3812?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13200976#comment-13200976 ]
Hadoop QA commented on MAPREDUCE-3812:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12513379/MAPREDUCE-3812-20120205.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1781//console
This message is automatically generated.
> Change default memory slot sizes to be 1.5GB
> --------------------------------------------
>
> Key: MAPREDUCE-3812
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3812
> Project: Hadoop Map/Reduce
> Issue Type: Sub-task
> Components: mrv2, performance
> Affects Versions: 0.23.0
> Reporter: Vinod Kumar Vavilapalli
> Assignee: Vinod Kumar Vavilapalli
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3812-20120205.txt
>
>
> After a few performance improvements tracked at MAPREDUCE-3561, like MAPREDUCE-3511 and MAPREDUCE-3567, even a 100K maps job can also run within 1GB vmem. We earlier increased AM slot size from 1 slot to two slots to work around the issues with AM heap. Now that those are fixed, we should go back to 1GB.
> This is just a configuration change.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira