You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Arun C Murthy (JIRA)" <ji...@apache.org> on 2011/07/16 01:25:59 UTC
[jira] [Created] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Enhance CS to cap concurrently running jobs
-------------------------------------------
Key: MAPREDUCE-2697
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
Project: Hadoop Map/Reduce
Issue Type: Bug
Reporter: Arun C Murthy
Assignee: Arun C Murthy
Fix For: 0.23.0
Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097284#comment-13097284 ]
Arun C Murthy commented on MAPREDUCE-2697:
------------------------------------------
Forgot to add - the crux of the solution is that we limit the amount of 'overhead' for applications in the cluster to a percent defined by yarn.capacity-scheduler.maximum-am-resource-percent. Thus, the #concurrent apps scales well with cluster capacity and #apps don't have to 'configured' by admins.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13098208#comment-13098208 ]
Hudson commented on MAPREDUCE-2697:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #808 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/808/])
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running applications per-queue & per-user.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/Queue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Attachment: MAPREDUCE-2697.patch
Updated patch, including unit tests.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097294#comment-13097294 ]
Hudson commented on MAPREDUCE-2697:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #907 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/907/])
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running applications per-queue & per-user.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/Queue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Status: Patch Available (was: Open)
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097260#comment-13097260 ]
Hadoop QA commented on MAPREDUCE-2697:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493055/MAPREDUCE-2697.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in hadoop-mapreduce-project.
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/609//console
This message is automatically generated.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097290#comment-13097290 ]
Hudson commented on MAPREDUCE-2697:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #830 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/830/])
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running applications per-queue & per-user.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/Queue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-2697:
-----------------------------------------------
Component/s: mrv2
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Attachment: MAPREDUCE-2697.patch
Updated patch to include new config in capacity-scheduler.xml, no code changes.
Also, default value of yarn.capacity-scheduler.maximum-am-resource-percent is 0.1, thus no more than 10% of the cluster is taken up by AMs and hence caps the #concurrent-apps.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Attachment: MAPREDUCE-2697.patch
Preliminary patch for review while I continue to test.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097293#comment-13097293 ]
Hadoop QA commented on MAPREDUCE-2697:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493062/MAPREDUCE-2697.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 11 new or modified tests.
-1 patch. The patch command could not apply the patch.
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/610//console
This message is automatically generated.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097298#comment-13097298 ]
Hudson commented on MAPREDUCE-2697:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #840 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/840/])
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running applications per-queue & per-user.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/Queue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Status: Patch Available (was: Open)
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097087#comment-13097087 ]
Hadoop QA commented on MAPREDUCE-2697:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12493013/MAPREDUCE-2697.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 8 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 13 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestLeafQueue
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestParentQueue
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-api.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-resourcemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-yarn-server-nodemanager.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-hs.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-shuffle.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//artifact/trunk/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-jobclient.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/606//console
This message is automatically generated.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Attachment: MAPREDUCE-2697.patch
Fixed failures caused due to low limits on activated apps.
Findbugs warnings seem unrelated - tracked in MAPREDUCE-2908.
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13097942#comment-13097942 ]
Hudson commented on MAPREDUCE-2697:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #785 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/785/])
MAPREDUCE-2697. Enhance CapacityScheduler to cap concurrently running applications per-queue & per-user.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1165403
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-api/src/main/java/org/apache/hadoop/yarn/api/records/impl/pb/ContainerIdPBImpl.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerContext.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/ParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/Queue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/resources/capacity-scheduler.xml
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestLeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestParentQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestUtils.java
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2697) Enhance CS to cap concurrently
running jobs
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2697?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2697:
-------------------------------------
Status: Open (was: Patch Available)
Checking the test failures...
> Enhance CS to cap concurrently running jobs
> -------------------------------------------
>
> Key: MAPREDUCE-2697
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2697
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Arun C Murthy
> Assignee: Arun C Murthy
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2697.patch, MAPREDUCE-2697.patch
>
>
> Enhance CS to cap concurrently running jobs ala 0.20.203
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira