You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (Created) (JIRA)" <ji...@apache.org> on 2012/01/17 19:58:39 UTC
[jira] [Created] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Capacity scheduler LeafQueues maximum capacity calculation issues
-----------------------------------------------------------------
Key: MAPREDUCE-3683
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Priority: Blocker
In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-3683:
----------------------------------------
Assignee: Arun C Murthy
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191556#comment-13191556 ]
Nathan Roberts commented on MAPREDUCE-3683:
-------------------------------------------
Let me clarify that last comment. Is there a specific reason we need to distinguish between a setting of 100% and UNDEFINED? I'm just wondering because if we're not benefiting from the distinction, it's adding confusion:
* Log messages of the form: "absoluteMaxCapacity = 3.4028235E38 [= Float.MAX_VALUE if maximumCapacity undefined, (parentAbsoluteCapacity * maximumCapacity) / 100 otherwise ]"
* webservices and webui both report 100% in the case where it's UNDEFINED
* Code like the following in a few places:
{code}
float absoluteMaxCapacity =
(Math.round(maximumCapacity * 100) ==
CapacitySchedulerConfiguration.UNDEFINED) ?
Float.MAX_VALUE :
(parent.getAbsoluteMaximumCapacity() * maximumCapacity);
{code}
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192632#comment-13192632 ]
Hadoop QA commented on MAPREDUCE-3683:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511729/MAPREDUCE-3683.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 9 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1662//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1662//console
This message is automatically generated.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193177#comment-13193177 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #408 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/408/])
Merge -c 1235858 from trunk to branch-0.23 to fix MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235860
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193182#comment-13193182 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1656 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1656/])
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235858
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193193#comment-13193193 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #433 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/433/])
Merge -c 1235858 from trunk to branch-0.23 to fix MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235860
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193786#comment-13193786 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #937 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/937/])
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235858
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193178#comment-13193178 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1583 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1583/])
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235858
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193191#comment-13193191 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1600 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1600/])
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235858
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192206#comment-13192206 ]
Nathan Roberts commented on MAPREDUCE-3683:
-------------------------------------------
Sorry if I'm just being dense here..
bq. 100% is all of parent's capacitY, UNDEFINED is effectively unlimited (infinity i.e. Float.MAX_VALUE).
I don't see how that lines up with previous comments:
{quote}
Tom: If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
Arun: That isn't true. It just means there is no max-cap for the child and it can use as much capacity as the parent queue gets, which might be capped by the parent/grandparent capacity/max-capacity and so on. This seems like the correct behaviour. Right?
{quote}
I think it works the way this last exchange describes it. Given the following queue configuration (cap,maxcap):
a (25,25)
b (75,75)
a.a1 (50,UNDEFINED)
a.a1 gets 25% of the cluster and nothing more.
I believe it works this way because the resources in the 'a' subtree are exhausted when it's trying to assign containers to that subtree(ParentQueue.assignToQueue).
I actually think it's correct that a leaf queue should not be able to break out of the limits set by its ancestors. However, if this is the case then UNDEFINED is the same as 100%.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Status: Open (was: Patch Available)
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193828#comment-13193828 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #970 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/970/])
MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235858
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189395#comment-13189395 ]
Nathan Roberts commented on MAPREDUCE-3683:
-------------------------------------------
My feeling is that an undefined max-capacity should be the same as a max-capacity of 100%. Then it would follow that a child's max-capacity would be the product of its ancestors' max-capacity values. This would be very similar to how absolutecapacity is calculated and I think that consistency would help reduce confusion.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191980#comment-13191980 ]
Hadoop QA commented on MAPREDUCE-3683:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511641/MAPREDUCE-3683.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.webapp.TestRMWebServicesCapacitySched
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1658//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1658//console
This message is automatically generated.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191966#comment-13191966 ]
Arun C Murthy commented on MAPREDUCE-3683:
------------------------------------------
[~dheeren] Yes, that's always the case. maxCap for a queue just prevents the queue itself.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193170#comment-13193170 ]
Arun C Murthy commented on MAPREDUCE-3683:
------------------------------------------
Thanks for the review Nathan. I'll fix the logging nit in the commit.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Status: Patch Available (was: Open)
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192269#comment-13192269 ]
Arun C Murthy commented on MAPREDUCE-3683:
------------------------------------------
[~nroberts] the interesting case was:
a (25, UNDEFINED)
a.a1 (50, UNDEFINED) v/s a.a1 (50, 100)
In which case, given the previous definition of max-cap (i.e. prior to this patch) this worked very differently... since max-cap was based on parent cap rather than parent max-cap.
However, given the current change, that distinction might not be necessary. We can fix that as a separate jira to unblock this bug, makes sense?
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189306#comment-13189306 ]
Thomas Graves commented on MAPREDUCE-3683:
------------------------------------------
{quote}
However, if the parent's max-capacity is undefined i.e. UNLIMITED (-1), we have two choices:
1. Use the parent's 'current' capacity to compute the max-capacity of the child at runtime
2. Set the child's max-capacity to be UNLIMITED too.
I'm leaning towards (2) since it's simpler and easier to explain to users i.e. it's more consistent. Otherwise, we run the risk of differing max-capacity (due to parent's capacity changing) which seems unpredictable and hard to explain to users/admins.
Thoughts?
{quote}
I guess I expected if the parent wasn't set (its max-capacity defaults to -1), which I thought implies a queue can use complete capacity of the cluster, so if a childs max-capacity was set, then it would just use that childs max-capacity - which would be % of the entire cluster. If you go with option 2 that means that if I set my max-capacity for my child queue it basically does nothing, which isn't what I would expect either.
{quote}
That isn't true. It just means there is no max-cap for the child and it can use as much capacity as the parent queue gets, which might be capped by the parent/grandparent capacity/max-capacity and so on. This seems like the correct behaviour. Right?
{quote}
Ok, sorry I was wrong, it does enforce it, what threw me off here was the web UI shows the absolute maximum capacity as 100% for the leaf queue and I expected this to be parents absolute max capacity. But I think that must be a web UI bug. I will file separate issue for that.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Attachment: MAPREDUCE-3683.patch
Thanks for the f/b Nathan. I've incorporated your comments.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193798#comment-13193798 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #150 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/150/])
Merge -c 1235858 from trunk to branch-0.23 to fix MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235860
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193816#comment-13193816 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #172 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/172/])
Merge -c 1235858 from trunk to branch-0.23 to fix MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235860
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191965#comment-13191965 ]
Arun C Murthy commented on MAPREDUCE-3683:
------------------------------------------
[~nroberts] 100% is all of parent's capacitY, UNDEFINED is effectively unlimited (infinity i.e. Float.MAX_VALUE).
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Status: Patch Available (was: Open)
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Dheeren Beborrtha (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191621#comment-13191621 ]
Dheeren Beborrtha commented on MAPREDUCE-3683:
----------------------------------------------
If capacity=maximumCapacity for one of the container queues, is this capapcity up for grab/stealing if other queues need it and this queue is unutilized?
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192298#comment-13192298 ]
Nathan Roberts commented on MAPREDUCE-3683:
-------------------------------------------
I'm fine with doing things in a separate jira, but doesn't this whole issue get much simpler if we can get rid of of MAX_FLOAT?
For example, my understanding of the current patch is that a configuration like:
a (25,UNDEFINED)
a.a1 (50, 50)
will let a job submitted to a.a1 consume the entire grid. This doesn't seem intuitive to me in that we ignored the maxcap setting of a.a1. Since the parent is UNDEFINED, computeAbsoluteMaxCapacity will return Float.MAX_VALUE. Am I missing something there?
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Attachment: MAPREDUCE-3683.patch
Updated with tests.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Attachment: MAPREDUCE-3683.patch
Straight-fwd patch for review.
Note that I had to special case absMaxCap for 'root'.
Still testing, will need to rebase after MAPREDUCE-3681.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13188957#comment-13188957 ]
Arun C Murthy commented on MAPREDUCE-3683:
------------------------------------------
bq. leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity - I think it should be using the parents absoluteMaximumCapacity.
Good catch. I'll fix this.
However, if the parent's max-capacity is undefined i.e. UNLIMITED (-1), we have two choices:
# Use the parent's 'current' capacity to compute the max-capacity of the child at runtime
# Set the child's max-capacity to be UNLIMITED too.
I'm leaning towards (2) since it's simpler and easier to explain to users i.e. it's more consistent. Otherwise, we run the risk of differing max-capacity (due to parent's capacity changing) which seems unpredictable and hard to explain to users/admins.
Thoughts?
----
bq. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
That isn't true. It just means there is no max-cap for the child and it can use as much capacity as the parent queue gets, which might be capped by the parent/grandparent capacity/max-capacity and so on. This seems like the correct behaviour. Right?
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193180#comment-13193180 ]
Hudson commented on MAPREDUCE-3683:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #417 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/417/])
Merge -c 1235858 from trunk to branch-0.23 to fix MAPREDUCE-3683. Fixed maxCapacity of queues to be product of parent maxCapacities.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235860
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSQueueUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestQueueParsing.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/webapp/TestRMWebServicesCapacitySched.java
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Fix Version/s: 0.23.1
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Nathan Roberts (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193084#comment-13193084 ]
Nathan Roberts commented on MAPREDUCE-3683:
-------------------------------------------
+1 on the patch
One question though, what should the default for capacity be? Right now I think it's UNDEFINED, which will throw an exception. If you decide to tweak that, the debug message in getCapacity() says "setCapacity", probably just remove it.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3683) Capacity scheduler LeafQueues
maximum capacity calculation issues
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3683?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3683:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Capacity scheduler LeafQueues maximum capacity calculation issues
> -----------------------------------------------------------------
>
> Key: MAPREDUCE-3683
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3683
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
> Attachments: MAPREDUCE-3683.patch, MAPREDUCE-3683.patch, MAPREDUCE-3683.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't take into account its parents maximum capacity when calculate its own maximum capacity, instead it seems to use the parents capacity. Looking at the code its using the parents absoluteCapacity and I think it should be using the parents absoluteMaximumCapacity.
> It also seems to only use the parents capacity in the leaf queues max capacity calculation when the leaf queue has a max capacity configured. If the leaf queues maximum-capacity is not configured, then it can use 100% of the cluster.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira