You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2012/06/05 14:15:23 UTC
[jira] [Created] (MAPREDUCE-4316) Improve default config values for
YARN
Arun C Murthy created MAPREDUCE-4316:
----------------------------------------
Summary: Improve default config values for YARN
Key: MAPREDUCE-4316
URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
Project: Hadoop Map/Reduce
Issue Type: Improvement
Affects Versions: 2.0.0-alpha
Reporter: Arun C Murthy
Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4316) Improve default config values
for YARN
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400235#comment-13400235 ]
Arun C Murthy commented on MAPREDUCE-4316:
------------------------------------------
If the default is too low, there are too many containers on a node for MR or other applications - we could land up with 8*16 containers on a 16G box which is WAY too high.
I missed MAPREDUCE-3812 but that was a bad change to goto 128M.
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Labels: performance
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4316) Improve default config values
for YARN
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13404678#comment-13404678 ]
Hadoop QA commented on MAPREDUCE-4316:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12534142/MAPREDUCE-4316.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
-1 eclipse:eclipse. The patch failed to build with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-common hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2538//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2538//console
This message is automatically generated.
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Assignee: Harsh J
> Labels: performance
> Attachments: MAPREDUCE-4316.patch
>
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4316) Improve default config values
for YARN
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13289528#comment-13289528 ]
Harsh J commented on MAPREDUCE-4316:
------------------------------------
I'd lowered the min-alloc via MAPREDUCE-3812. But had left the MR defaults for job allocs the same as they were. How does that min-alloc value lower the performance though Arun?
What ought one raise - request size or rely on min-alloc? I was under the impression that min-allocs should be low to grant lower requests per container, but large requests (from MR App) should anyway do fine as they did before. Which one ought users rely on?
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4316) Improve default config values for
YARN
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4316:
-------------------------------
Assignee: Harsh J
Release Note: Raise yarn.scheduler.minimum.allocation-mb to 512 MB (Previous minimum was 128 MB)
Status: Patch Available (was: Open)
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Assignee: Harsh J
> Labels: performance
> Attachments: MAPREDUCE-4316.patch
>
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-4316) Improve default config values
for YARN
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13400151#comment-13400151 ]
Harsh J commented on MAPREDUCE-4316:
------------------------------------
Arun - Ping? Comments on my above questions? Should we really raise the defaults?
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Labels: performance
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4316) Improve default config values for
YARN
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4316:
-------------------------------
Component/s: scheduler
resourcemanager
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Labels: performance
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-4316) Improve default config values for
YARN
Posted by "Harsh J (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-4316?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Harsh J updated MAPREDUCE-4316:
-------------------------------
Attachment: MAPREDUCE-4316.patch
Right, but MR provides its own set of default resource requests for both AM and Task containers, none of which changed from what they were. So for Trevor's issue at http://s.apache.org/avd, I doubt if the issue were the low min-alloc as you say. Or am I still mistaken and the min-alloc is an enforcer somehow (meaning, MR AM is broken in requesting proper resources?).
However, you're right that it affects other apps that would not make an explicit request for a specific value, and, here's a patch that bumps it to 512 MB for min-alloc.
> Improve default config values for YARN
> --------------------------------------
>
> Key: MAPREDUCE-4316
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-4316
> Project: Hadoop Map/Reduce
> Issue Type: Improvement
> Components: resourcemanager, scheduler
> Affects Versions: 2.0.0-alpha
> Reporter: Arun C Murthy
> Labels: performance
> Attachments: MAPREDUCE-4316.patch
>
>
> Currently some of our configs are way off e.g. min-alloc is 128M while max-alloc is 10240.
> This leads to poor out-of-box performance as noticed by some users: http://s.apache.org/avd
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira