You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (Created) (JIRA)" <ji...@apache.org> on 2012/02/22 15:17:48 UTC
[jira] [Created] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
-----------------------------------------------------------------------------------------------------------------------
Key: MAPREDUCE-3893
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420917#comment-13420917 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #2577 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/2577/])
MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364764)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364764
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421360#comment-13421360 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #323 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/323/])
svn merge -c 1364764 FIXES: MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364766)
Result = UNSTABLE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364766
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421373#comment-13421373 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #1114 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/1114/])
MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364764)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364764
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3893:
----------------------------------
Attachment: MAPREDUCE-3893-1.txt
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13222105#comment-13222105 ]
Arun C Murthy commented on MAPREDUCE-3893:
------------------------------------------
Also, for small clusters we should cut down max-system-apps to 100 or so.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420893#comment-13420893 ]
Robert Joseph Evans commented on MAPREDUCE-3893:
------------------------------------------------
The latest set of changes look good to me +1.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13228430#comment-13228430 ]
Thomas Graves commented on MAPREDUCE-3893:
------------------------------------------
I think its more of being able to configure it differently per queue and get better utilization then stopping deadlocks (Although it could also work around deadlock issues). Right now there is only one setting so if you have queues that are running very different jobs, its not very efficient. For instance, what if one queue is running all uberAM's, You want that queue to have maximum-am-resource-percent close to 100%, but if another queue on the same cluster is running jobs with lots of maps/reduces per job you may only want it at 10% because starting other AM's will just waste memory as they will have to wait for the first job anyway. So lets say you set it at 50% to get somewhere in the middle. That means you are wasting half of the queue that wants to run all uberAM's. That 50% may also be to high if you have yet another queue that is set to use 1% of the cluster capacity. Note that the calculation for maxActiveApps is using the minimumAllocation setting, so if a user sets its AM memory to be much more then that and the queue is small, you could deadlock again.
Note that I'm mostly interested in maximum-am-resource-percent. The maximum-applications we can leave alone.
I believe the way Eric has it only makes it more complicated if you want to configure it per queue. The global is still there but if you want to override then you can set it per queue. Obviously more settings does introduce the chance for more mis-config but I don't see it anymore complicated then what is already there per queue. I'm worried without this we won't be using the cluster very efficiently or we will run into deadlocks because we set it high but the smaller capacity queues won't be able to handle that.
If you have other ideas or thoughts please let us know.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13246398#comment-13246398 ]
Thomas Graves commented on MAPREDUCE-3893:
------------------------------------------
Arun, any input on this?
On the configs - would it be better to remove the global setting and just have this per queue, that way it would match the other queue configs. I'm not sure if that is considered breaking api compatibility though.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-3893:
-------------------------------------
Target Version/s: 0.23.3, 3.0.0, 2.2.0-alpha (was: 0.23.3, 2.0.0-alpha, 3.0.0)
Status: Patch Available (was: Open)
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13221360#comment-13221360 ]
Hadoop QA commented on MAPREDUCE-3893:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12516901/MAPREDUCE-3893-1.txt
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1990//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1990//console
This message is automatically generated.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3893:
----------------------------------
Status: Patch Available (was: Open)
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-3893:
-------------------------------------
Attachment: MAPREDUCE-3893.patch
patch upmerged to trunk. Fixed a couple issues with maxApplications per queue settings, added documentation, add more tests.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3893:
-------------------------------------
Status: Open (was: Patch Available)
Eric, I'm not convinced that making it configurable per-queue will solve the deadlock problem. I'm worried making it more complicated will mean more room for error i.e. you could introduce more deadlocks by being over-aggressive with #applications in a queue due to mis-config leading to more deadlocks.
Could you please share the configs you saw which led to the deadlock? Thanks!
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves reassigned MAPREDUCE-3893:
----------------------------------------
Assignee: Thomas Graves (was: Eric Payne)
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Closed] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy closed MAPREDUCE-3893.
------------------------------------
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 2.0.2-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Eric Payne (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne updated MAPREDUCE-3893:
----------------------------------
Priority: Critical (was: Major)
I am marking this as critical because the deadlock situation is a real possibility when one queue is dedicated to one user that submits a large number of jobs simultaneously in a currently quiet cluster. We have a particular use case (GDM) that encounters this situation rebularly.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Priority: Critical
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Eric Payne (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eric Payne reassigned MAPREDUCE-3893:
-------------------------------------
Assignee: Eric Payne
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13421429#comment-13421429 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #1146 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/1146/])
MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364764)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364764
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419336#comment-13419336 ]
Hadoop QA commented on MAPREDUCE-3893:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12537364/MAPREDUCE-3893.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 1 new or modified test files.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 4 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2636//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2636//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/2636//console
This message is automatically generated.
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420913#comment-13420913 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #2512 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/2512/])
MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364764)
Result = SUCCESS
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364764
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Thomas Graves (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13419386#comment-13419386 ]
Thomas Graves commented on MAPREDUCE-3893:
------------------------------------------
findbugs are unrelated - see https://issues.apache.org/jira/browse/MAPREDUCE-4452
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Robert Joseph Evans (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3893:
-------------------------------------------
Target Version/s: 0.23.3, 2.0.0, 3.0.0 (was: 0.23.2)
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Eric Payne
> Priority: Critical
> Attachments: MAPREDUCE-3893-1.txt
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3893) allow capacity scheduler configs
maximum-applications and maximum-am-resource-percent configurable on a per
queue basis
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated MAPREDUCE-3893:
-------------------------------------------
Resolution: Fixed
Fix Version/s: 2.2.0-alpha
3.0.0
0.23.3
Status: Resolved (was: Patch Available)
Thanks Tom,
I put this into trunk, branch-2, and branch-0.23
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3893) allow capacity scheduler
configs maximum-applications and maximum-am-resource-percent configurable
on a per queue basis
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3893?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13420941#comment-13420941 ]
Hudson commented on MAPREDUCE-3893:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #2533 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/2533/])
MAPREDUCE-3893. allow capacity scheduler configs max-apps and max-am-pct per queue (tgraves via bobby) (Revision 1364764)
Result = FAILURE
bobby : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1364764
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-site/src/site/apt/CapacityScheduler.apt.vm
> allow capacity scheduler configs maximum-applications and maximum-am-resource-percent configurable on a per queue basis
> -----------------------------------------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-3893
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3893
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Thomas Graves
> Priority: Critical
> Fix For: 0.23.3, 3.0.0, 2.2.0-alpha
>
> Attachments: MAPREDUCE-3893-1.txt, MAPREDUCE-3893.patch
>
>
> The capacity scheduler configs for maximum-applications and maximum-am-resource-percent are currently configured globally and then made proportional to each queue based on its capacity. There are times when this may not work well. some exampless - if you have a queue that is running on uberAM jobs, the jobs a queue is running always has a small number of containers, and then you have the opposite where in a queue with very small capacity, you may want to limit the am resources even more so you don't end up deadlocked with all your capacity being used for app masters.
> I think we should make those configurable on a per queue basis.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira