You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-issues@hadoop.apache.org by "Ahmed Radwan (JIRA)" <ji...@apache.org> on 2011/08/09 02:08:27 UTC
[jira] [Created] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
-----------------------------------------------------------------------------------------
Key: MAPREDUCE-2788
URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
Project: Hadoop Map/Reduce
Issue Type: Bug
Components: mrv2
Reporter: Ahmed Radwan
Assignee: Ahmed Radwan
The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Status: Patch Available (was: Open)
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128295#comment-13128295 ]
Hadoop QA commented on MAPREDUCE-2788:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499138/MAPREDUCE-2788_rev4.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1032//console
This message is automatically generated.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13123872#comment-13123872 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Ahmed - the LeafQueue is interminably tied to CS and hence doing it in CS.allocate is reasonable.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Status: Patch Available (was: Open)
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Attachment: MAPREDUCE-2788_rev2.patch
Thanks Arun, I have updated the patch accordingly. Using CS.normalizeRequest, the request's memory is now set to minMemory if it is zero (avoiding the divide by zero). The revised patch is based on the recent version (after merge to trunk).
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127410#comment-13127410 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Arun, Yes I am working on it. Planning to post an update later today. Thanks!
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13089647#comment-13089647 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Thanks Arun! The LeafQueue uses the CapacitySchedulerContext interface (CS is an implementation), so other implementations can be added. We don't need to tie the LeafQueue to the CS implementation. So I think the LeafQueue.assignContainer() should check/ensure that the request has no zero memory request before dividing by zero. In my last patch I added normalizeRequest() to the interface so any other implementations are enforced to implement this method (which takes care of this zero memory case). What do you think?
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124781#comment-13124781 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Thanks Arun, will take a look.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131571#comment-13131571 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Build #45 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Build/45/])
Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13082821#comment-13082821 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Ahmed - another way would be to ensure there are no requests which need memory=0 - this check can be enforced in CS.normalizeRequests. Do you want to take a crack at that? Also, please mark this as PA when done - that way I (or other committers) can pick it up on my PA filter. Thanks.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130977#comment-13130977 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Hdfs-0.23-Commit #25 (See [https://builds.apache.org/job/Hadoop-Hdfs-0.23-Commit/25/])
Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13124782#comment-13124782 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Thanks Ahmed.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Status: Patch Available (was: Open)
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Attachment: MAPREDUCE-2788_rev3.patch
Thanks Arun for your comments, I have looked into normalizing the requests within the CapacityScheduler.
It doesn't seem that the call to LeafQueue.assignContainer(..) come via CapacityScheduler.allocate(). It gets called through the call path:
LeafQueue.assignContainer(..) <- assignNodeLocalContainers(..) <-- LeafQueue.assignContainersOnNode(..) <- LeafQueue.assignContainers(..)
There are alternative paths, but all lead to the same source.
The SchedulerApp application (in the LeafQueue.assignContainers(..) call) is one of the Map<ApplicationAttemptId, SchedulerApp> applicationsMap values. This applicationsMap is only populated through LeafQueue.addApplication(..).
The LeafQueue.addApplication(..) is called through the path: LeafQueue.addApplication(..) <- LeafQueue.submitApplication(..) <- CapacityScheduler.addApplication(..).
So I have added code to CapacityScheduler.addApplication(..) to normalize all resource requests for the SchedulerApp before submitting to the queue.
If the LeafQueue is interminably tied to CS, we may need to update the references in LeafQueue to use CapacityScheduler instead of CapacitySchedulerContext, this will make such dependency clear and avoid future confusions. I haven't made this interface change in the attached patch, as it requires more changes to other components, but if we agree about it, I can do it in a following issue.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Priority: Major (was: Critical)
Summary: Normalize requests in FifoScheduler.allocate to prevent NPEs later (was: LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0)
Changing the title to reflect the existing bug.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Attachment: MAPREDUCE-2788.patch
The attached patch avoids the described crash and returns Resources.none() in case memory==0.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130983#comment-13130983 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Hdfs-trunk-Commit #1190 (See [https://builds.apache.org/job/Hadoop-Hdfs-trunk-Commit/1190/])
MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this. Thanks Ahmed!
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Attachment: MAPREDUCE-2788_rev5.patch
Added a simple test case to verify the request normalization behavior for different valid and invalid memory values.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Priority: Critical (was: Major)
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130980#comment-13130980 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Common-trunk-Commit #1111 (See [https://builds.apache.org/job/Hadoop-Common-trunk-Commit/1111/])
MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Attachment: MAPREDUCE-2788_rev4.patch
The attached new patch pulls out requests' normalization methods to SchedulerUtils, and both FS and CS now use them. FS.allocate() already calls normalizeRequests(), but the previous implementation for normalizeRequest() in FS doesn't rectify the case where memory==0. This is now fixed. Please let me know if you have further comments.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Ahmed Radwan updated MAPREDUCE-2788:
------------------------------------
Status: Patch Available (was: Open)
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13129902#comment-13129902 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Will do, thanks Mahadev.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127982#comment-13127982 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Yes CS.allocate() already calls normalizeRequests(..), no need to change anything there. But the call scenario described in my previous comment is the one causing the problem, not CS.allocate(..), it is the CS.addApplication(..).
The latest patch fix to CS.addApplication(..) to also normalize requests is the one required. What do you think?
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131608#comment-13131608 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Build #57 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Build/57/])
Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131008#comment-13131008 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Mapreduce-trunk-Commit #1130 (See [https://builds.apache.org/job/Hadoop-Mapreduce-trunk-Commit/1130/])
MAPREDUCE-2788. Normalize resource requests in FifoScheduler appropriately. Contributed by Ahmed Radwan.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186467
Files :
* /hadoop/common/trunk/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/trunk/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13131007#comment-13131007 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Mapreduce-0.23-Commit #26 (See [https://builds.apache.org/job/Hadoop-Mapreduce-0.23-Commit/26/])
Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Status: Open (was: Patch Available)
Ahmed - not sure when this happened, but looks like CS.allocate already normalizes the incoming request.
Thus, we might only need to fix FifoScheduler.normalizeRequest?
We should pull out normalizeRequest to SchedulerUtils and use it from both places?
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Status: Open (was: Patch Available)
Ahmed - sorry I wasn't clear. I was thinking along hte lines of ensuring all incoming requests (via CS.allocate) are normalized to ensure LeafQueue doesn't have to worry about normalizing them. Thoughts?
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130518#comment-13130518 ]
Hadoop QA commented on MAPREDUCE-2788:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499662/MAPREDUCE-2788_rev5.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 2 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 160 new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//testReport/
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-common.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-app.html
Findbugs warnings: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//artifact/trunk/hadoop-mapreduce-project/patchprocess/newPatchFindbugsWarningshadoop-mapreduce-client-core.html
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1067//console
This message is automatically generated.
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Arun C Murthy (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Arun C Murthy updated MAPREDUCE-2788:
-------------------------------------
Fix Version/s: 0.23.0
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127969#comment-13127969 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Sorry, I missed the fact that CS seems to be fine now... a while since I looked at it. Apologies!
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128066#comment-13128066 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Ahmed, when we add a new SchedulerApp it doesn't have any resource requests until the ApplicationsManager (or later, the ApplicationMaster itself) calls CS.allocate. Hence it isn't necessary to normalize requests. Makes sense?
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13130986#comment-13130986 ]
Hudson commented on MAPREDUCE-2788:
-----------------------------------
Integrated in Hadoop-Common-0.23-Commit #25 (See [https://builds.apache.org/job/Hadoop-Common-0.23-Commit/25/])
Merge -c 1186467 from trunk to branch-0.23 to complete fix for MAPREDUCE-2788.
acmurthy : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1186468
Files :
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/CHANGES.txt
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/SchedulerUtils.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/fifo/FifoScheduler.java
* /hadoop/common/branches/branch-0.23/hadoop-mapreduce-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/test/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/TestSchedulerUtils.java
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Fix For: 0.23.0
>
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch, MAPREDUCE-2788_rev5.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Ahmed Radwan (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13128136#comment-13128136 ]
Ahmed Radwan commented on MAPREDUCE-2788:
-----------------------------------------
Yes, it makes sense now. Thanks
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (MAPREDUCE-2788) Normalize requests in
FifoScheduler.allocate to prevent NPEs later
Posted by "Mahadev konar (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Mahadev konar updated MAPREDUCE-2788:
-------------------------------------
Status: Open (was: Patch Available)
Ahmed,
Could you please add a test case with this patch?
> Normalize requests in FifoScheduler.allocate to prevent NPEs later
> ------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch, MAPREDUCE-2788_rev4.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Hadoop QA (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127800#comment-13127800 ]
Hadoop QA commented on MAPREDUCE-2788:
--------------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12499089/MAPREDUCE-2788_rev3.patch
against trunk revision .
+1 @author. The patch does not contain any @author tags.
-1 tests included. The patch doesn't appear to include any new or modified tests.
Please justify why no new tests are needed for this patch.
Also please list what manual steps were performed to verify this patch.
-1 javadoc. The javadoc tool appears to have generated 20 warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
+1 findbugs. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed unit tests in .
+1 contrib tests. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//testReport/
Console output: https://builds.apache.org/job/PreCommit-MAPREDUCE-Build/1023//console
This message is automatically generated.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch, MAPREDUCE-2788_rev3.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (MAPREDUCE-2788) LeafQueue.assignContainer() can
cause a crash if request.getCapability().getMemory() == 0
Posted by "Arun C Murthy (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/MAPREDUCE-2788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13127321#comment-13127321 ]
Arun C Murthy commented on MAPREDUCE-2788:
------------------------------------------
Ahmed, do you think you'll get to this soon? Thanks.
> LeafQueue.assignContainer() can cause a crash if request.getCapability().getMemory() == 0
> -----------------------------------------------------------------------------------------
>
> Key: MAPREDUCE-2788
> URL: https://issues.apache.org/jira/browse/MAPREDUCE-2788
> Project: Hadoop Map/Reduce
> Issue Type: Bug
> Components: mrv2
> Reporter: Ahmed Radwan
> Assignee: Ahmed Radwan
> Priority: Critical
> Attachments: MAPREDUCE-2788.patch, MAPREDUCE-2788_rev2.patch
>
>
> The assignContainer() method in org.apache.hadoop.yarn.server.resourcemanager.scheduler.capacity.LeafQueue can cause the scheduler to crash if the ResourseRequest capability memory == 0 (divide by zero).
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira