You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Siddharth Seth (Created) (JIRA)" <ji...@apache.org> on 2012/01/24 02:55:39 UTC
[jira] [Created] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Incorrect headroom reported to jobs
-----------------------------------
Key: MAPREDUCE-3713
URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2, resourcemanager
Affects Versions: 0.23.0
Reporter: Siddharth Seth
Priority: Blocker
With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Attachment: MAPREDUCE-3713.patch
Updated with unit tests.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3713:
-----------------------------------------------
Fix Version/s: 0.23.1
Assignee: Arun C Murthy
This is blocking gridmix..
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Status: Open (was: Patch Available)
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193199#comment-13193199 ]
Hadoop QA commented on MAPREDUCE-3713:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511858/MAPREDUCE-3713.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1671//console
This message is automatically generated.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193784#comment-13193784 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk #937 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk/937/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235989
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193459#comment-13193459 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #436 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/436/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
svn merge --ignore-ancestry -c 1235989 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193452#comment-13193452 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1586 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1586/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235989
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192934#comment-13192934 ]
Hadoop QA commented on MAPREDUCE-3713:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511816/MAPREDUCE-3713.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 3 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.mapred.TestJobClientGetJob
org.apache.hadoop.mapred.TestMRWithDistributedCache
org.apache.hadoop.mapred.TestLocalModeWithNewApis
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1668//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1668//console
This message is automatically generated.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Status: Patch Available (was: Open)
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193454#comment-13193454 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #420 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/420/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
svn merge --ignore-ancestry -c 1235989 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193826#comment-13193826 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk #970 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk/970/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235989
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Attachment: MAPREDUCE-3713.patch
Removed some debug logging.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Vinod Kumar Vavilapalli (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193437#comment-13193437 ]
Vinod Kumar Vavilapalli commented on MAPREDUCE-3713:
----------------------------------------------------
+1. The patch looks good.
I guess writing a test specific to the bug in question is complicated, given the current test infrastructure. May be later.
The test failures are unrelated. I am pushing this in.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193450#comment-13193450 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #411 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/411/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
svn merge --ignore-ancestry -c 1235989 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Status: Patch Available (was: Open)
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Vinod Kumar Vavilapalli (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli updated MAPREDUCE-3713:
-----------------------------------------------
Resolution: Fixed
Release Note: Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application.
Hadoop Flags: Reviewed
Status: Resolved (was: Patch Available)
Just committed this to trunk and branch-0.23. Thanks Arun!
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193481#comment-13193481 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1603 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1603/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235989
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192742#comment-13192742 ]
Hadoop QA commented on MAPREDUCE-3713:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12511758/MAPREDUCE-3713.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 eclipse:eclipse. The patch built with eclipse:eclipse.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed these unit tests:
org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.TestApplicationLimits
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1665//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1665//console
This message is automatically generated.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Status: Patch Available (was: Open)
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Attachment: MAPREDUCE-3713.patch
Fixed headroom to take into a/c user consumption, not just app consumption.
Writing a unit test...
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-3713:
-------------------------------------
Status: Open (was: Patch Available)
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193796#comment-13193796 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #150 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/150/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
svn merge --ignore-ancestry -c 1235989 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193814#comment-13193814 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #172 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/172/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
svn merge --ignore-ancestry -c 1235989 ../../trunk/
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235990
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13192942#comment-13192942 ]
Arun C Murthy commented on MAPREDUCE-3713:
------------------------------------------
The test failures are due to LocalJobRunner tests failing and are completely unrelated to this patch.
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-3713) Incorrect headroom reported to
jobs
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-3713?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13193451#comment-13193451 ]
Hudson commented on MAPREDUCE-3713:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1659 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1659/])
MAPREDUCE-3713. Fixed the way head-room is allocated to applications by CapacityScheduler so that it deducts current-usage per user and not per-application. Contributed by Arun C Murthy.
vinodkv : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1235989
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerApp.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/LeafQueue.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/TestApplicationLimits.java
> Incorrect headroom reported to jobs
> -----------------------------------
>
> Key: MAPREDUCE-3713
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-3713
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2, resourcemanager
> Affects Versions: 0.23.0
> Reporter: Siddharth Seth
> Assignee: Arun C Murthy
> Priority: Blocker
> Fix For: 0.23.1
>
> Attachments: MAPREDUCE-3713.patch, MAPREDUCE-3713.patch, MAPREDUCE-3713.patch
>
>
> With multiple jobs submitted per user, and multiple users submitting jobs - the headroom reported to the AppMasters is incorrect (very high).
> Leads to a deadlock - reduces started, map tasks not complete... and reduces are not preempted by the AM due to the incorrect headroom.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira