[jira] [Commented] (MAPREDUCE-2723) MR-279: port MAPREDUCE-2324 to mrv2

["Robert Joseph Evans (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2723?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13069602#comment-13069602 ] 

Robert Joseph Evans commented on MAPREDUCE-2723:
------------------------------------------------

I have been looking at MR-279 and I want to do something similar to MR-2324 it is just not really set up to do it easily. The scheduling is split up between the resource manager and the application master. And in fact the resource manager/application master are completely ignoring disk utilization at this point.

The plan is to add in disk utilization to the resources that the RM uses, and then have AM request both disk and RAM space for reduces with disk space based off of the size estimate currently used. Then inside the scheduler, which is the right place in my opinion to decide if a request is being starved or not, it would do just what MR-2324 does now but more generalized for all resource constraints, not just disk. This means that all schedulers would have to be modified to support this, but I can make the code generic so it should be fairly simple to do. I just need to dig into the MR-279 code to decide exactly how I want to insert this in. I should hopefully have a patch by mid next week.


> MR-279: port MAPREDUCE-2324 to mrv2
> -----------------------------------
>
>                 Key: MAPREDUCE-2723
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2723
>             Project: Hadoop Map/Reduce
>          Issue Type: Sub-task
>          Components: mrv2
>    Affects Versions: 0.23.0
>            Reporter: Robert Joseph Evans
>            Assignee: Robert Joseph Evans
>             Fix For: 0.23.0
>
>
> MRV2 currently does not take reduce disk usage into account when trying to schedule a container.  For feature parity with the original map reduce it should be extended to allow for disk space requests within containers along with RAM requests.  We then also need to port MAPREDUCE-2324 to the scheduler to allow it to avoid starvation of containers that might never get the resources that they need.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira