You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Thomas Graves (Created) (JIRA)" <ji...@apache.org> on 2012/01/17 19:14:41 UTC
[jira] [Created] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
capacity scheduler LeafQueues calculate used capacity wrong
-----------------------------------------------------------
Key: MAPREDUCE-3681
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Affects Versions: 0.23.0
Reporter: Thomas Graves
Priority: Blocker
In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
In updateResource in LeafQueue:
setUsedCapacity(
usedResources.getMemory() / (clusterResource.getMemory() * capacity));
I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189022#comment-13189022 ]
Hadoop QA commented on MAPREDUCE-3681:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511101/MAPREDUCE-3681.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
-1 findbugs. The patch appears to introduce 2 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1629//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1629//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1629//console
This message is automatically generated.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191785#comment-13191785 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1593 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1593/])
MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235103
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192088#comment-13192088 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #935 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/935/])
MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235103
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3681:
-----------------------------------------------
Fix Version/s: 0.23.1
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191780#comment-13191780 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #400 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/400/])
Merge -c 1235103 from trunk to branch-0.23 to fix MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235104
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192110#comment-13192110 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #148 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/148/])
Merge -c 1235103 from trunk to branch-0.23 to fix MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235104
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191714#comment-13191714 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1649 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1649/])
MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235103
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Attachment: MAPREDUCE-3681.patch
Trivial patch.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191795#comment-13191795 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #425 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/425/])
Merge -c 1235103 from trunk to branch-0.23 to fix MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235104
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191782#comment-13191782 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #410 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/410/])
Merge -c 1235103 from trunk to branch-0.23 to fix MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235104
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189566#comment-13189566 ]
Hadoop QA commented on MAPREDUCE-3681:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511198/MAPREDUCE-3681.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1641//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1641//console
This message is automatically generated.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13189287#comment-13189287 ]
Thomas Graves commented on MAPREDUCE-3681:
------------------------------------------
I need to look more, but trying this patch out always seems to yield 0% used capacity (according to webui).
What is meant to be the difference between used capacity and utilization?
Also note that there is a big log statement in setupQueueConfigs, that prints the formulas on how these are calculated so we would want to change that to match.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191336#comment-13191336 ]
Arun C Murthy commented on MAPREDUCE-3681:
------------------------------------------
+1 good catch Thomas!
bq. root.a.a1 used = 4/(32 .5) = 25%, utilization = 4/(32.25) = 50% => this just happens to work because capacity = parent asoluteCap
I made the same mistake to verify my calculations...
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191575#comment-13191575 ]
Hadoop QA commented on MAPREDUCE-3681:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511466/MAPREDUCE-3681.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1653//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1653//console
This message is automatically generated.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192141#comment-13192141 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #968 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/968/])
MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235103
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Priority: Critical (was: Blocker)
Good catch Thomas.
Thankfully this isn't super-critical because 'usedCapacity' is used nowhere other than displaying queue-stats to clients i.e. it doesn't runtime behaviour of the CapacityScheduler.
I'll fix this.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Thomas Graves (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190842#comment-13190842 ]
Thomas Graves commented on MAPREDUCE-3681:
------------------------------------------
Thanks for the update Arun.
I was misunderstanding what usedCapacity meant, but I think I now understand, although I don't think the current formula for used Capacity works in all cases. Please correct me if I’m off on the definition here. I believe used Capacity is meant to be what % of the parent am I using. Similar to how capacity/max capacity apply to the parent. I expected usedCapacity to be what utilization is defined as, which is what I thought users will be interested in -> how much of the resources normally allocated to my queue (absoluteCapacity) am I using right now. I think Utilization = absoluteUsedCapacity.
Is a user normally really going to care what % of my parent queue they are using? To me the absoluteCapacity is what I’m interested in because my queue has X amount of memory allocated to it, and I have Y left.
I can see admins being interested in the used Capacity as they can quickly see what percent of the parent is being used by the various leaf queues.
right now definitions are:
utilization = usedResources.getMemory() / (clusterResource.getMemory() * absoluteCapacity)
usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() * capacity)
Here is an example, which I believe demonstrates usedCapacity being off - lets say cluster has 32G memory total:
Queue definitions:
root - capacity = absoluteCapacity = 100% - 32G
root.a - capacity=50%, absoluteCap = 100*50=50% - 16G
root.a.a1 - capacity = 50%, absoluteCap = (100*50)*50 = 25% - 8G
root.a.a2 - capacity = 50%, absoluteCap = (100*50)*50% = 25% - 8G
root.b - capacity = 20%, absoluteCap = 100*20=20% - 6.4G
root.b.b1 - capacity = 60%, absoluteCap = 100*20%*60% = 12%, MaxCap = -1 (lets me go over my capacity) = 3.84G
root.c - capacity = 30%, absoluteCap = 100*30=30% - 9.6G
root.c.c1 - capacity = 75%, absoluteCap = 100*30%*75* = 22.5% = 7.2G
Now I start a job that uses 4G submitted to a1.
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.a used = 4/(32*.5) = 25%, utilization = 4/(32*.5) = 25% = 4G of 16G
root.a.a1 used = 4/(32 *.5) = 25%, utilization = 4/(32*.25) = 50% => this just happens to work because capacity = parent asoluteCap
job in a1 finishes. now start a job that uses 4G submitted to b1
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.b used = 4/(32*.2) = 62.5%, utilization = 4/(32*.2) = 62.5% = 4G of 6.4G
root.b.b1 used = 4/(32 *.6) = 20.8% (4G of 6.4G ??), utilization = 4/(32*.12) = 104% (4G of 3.84G)
the 20.8% doesn't make sense to me? I'm using 4G of my parents 6.4G - that should be 62.5%.
job in a1 finishes. now start a job that uses 4G submitted to c1
root used = 4/(32 * 1) = 12.5%, utilization= 4/(32*1) = 12.5% = 4G of 32G
root.c used = 4/(32*.3) = 41.6%, utilization = 4/(32*.3) = 41.6% = 4G of 9.6G
root.c.c1 used = 4/(32 *.75) = 16.6% (4G of 9.6G ??), utilization = 4/(32*.225) = 55.5% (4G of 7.2G)
Again the 16.6% I think is wrong as it should be 4G of the 9.6G or 41.6%
I think the formula should be
usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() * parent.getAbsoluteCapacity())
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Status: Open (was: Patch Available)
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13190867#comment-13190867 ]
Hadoop QA commented on MAPREDUCE-3681:
--------------------------------------
+1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511466/MAPREDUCE-3681.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1650//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1650//console
This message is automatically generated.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13191715#comment-13191715 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1576 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1576/])
MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235103
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Attachment: MAPREDUCE-3681.patch
Findbugs caught the 'usedCapacity=0' problem... integral division! :)
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Thomas Graves (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Thomas Graves updated MAPREDUCE-3681:
-------------------------------------
Attachment: MAPREDUCE-3681.patch
I had made the updates for
usedCapacity = usedResources.getMemory() / (clusterResource.getMemory() * parent.getAbsoluteCapacity())
while testing, here is what I was running.
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Assigned) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy reassigned MAPREDUCE-3681:
----------------------------------------
Assignee: Arun C Murthy
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Blocker
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Status: Patch Available (was: Open)
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3681:
-------------------------------------
Status: Patch Available (was: Open)
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3681) capacity scheduler LeafQueues
calculate used capacity wrong
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3681?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192129#comment-13192129 ]
Hudson commented on MAPREDUCE-3681:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #170 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/170/])
Merge -c 1235103 from trunk to branch-0.23 to fix MAPREDUCE-3681. Fixed computation of queue's usedCapacity.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235104
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
> capacity scheduler LeafQueues calculate used capacity wrong
> -----------------------------------------------------------
>
> Key: MAPREDUCE-3681
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3681
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Affects Versions: 0.23.0
> Reporter: Thomas Graves
> Assignee: Arun C Murthy
> Priority: Critical
> Attachments: MAPREDUCE-3681.patch, MAPREDUCE-3681.patch, MAPREDUCE-3681.patch
>
>
> In the Capacity scheduler if you configure the queues to be hierarchical where you have root -> parent queue -> leaf queue, the leaf queue doesn't calculate the used capacity properly. It seems to be using the entire cluster memory rather then its parents memory capacity.
> In updateResource in LeafQueue:
> setUsedCapacity(
> usedResources.getMemory() / (clusterResource.getMemory() * capacity));
> I think the clusterResource.getMemory() should be something like getParentsMemory().
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira