[jira] [Commented] (MAPREDUCE-2694) AM releases too many containers due to the protocol

["Sharad Agarwal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/MAPREDUCE-2694?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13066782#comment-13066782 ] 

Sharad Agarwal commented on MAPREDUCE-2694:
-------------------------------------------

>> Yes, this could lead to some waste, but the system is eventually consistent.

There will be lot of waste. Particularly for applications which are ramping up the requests with time, and not putting all the requests upfront.
Another issue will be the *sub-optimal scheduling* in AM.
In the above example:
- AM asks for 3 additional containers(total 7) on different host H2. 
- The request table in AM will get overwritten with 4 on H1 and 3 on H2. 
- RM may allocate containers on H1 or H2. But in reality it should only try to assign on H2.
- If RM gives containers to AM on H1 first, AM will do off host assignments and will release the ones on H2.

> AM releases too many containers due to the protocol
> ---------------------------------------------------
>
>                 Key: MAPREDUCE-2694
>                 URL: https://issues.apache.org/jira/browse/MAPREDUCE-2694
>             Project: Hadoop Map/Reduce
>          Issue Type: Bug
>          Components: mrv2
>            Reporter: Arun C Murthy
>            Assignee: Arun C Murthy
>
> - AM sends request asking 4 containers on host H1.
> - Asynchronously, host H1 reaches RM and gets assigned 4 containers. RM at this point, sets the value against H1 to
> zero in its aggregate request-table for all apps.
> - In the mean-while AM gets to need 3 more containers, so a total of 7 including the 4 from previous request.
> - Today, AM sends the absolute number of 7 against H1 to RM as part of its request table.
> - RM seems to be overriding its earlier value of zero against H1 to 7 against H1. And thus allocating 7 more
> containers.
> - AM already gets 4 in this scheduling iteration, but gets 7 more, a total of 11 instead of the required 7.

--
This message is automatically generated by JIRA.
For more information on JIRA, see: http://www.atlassian.com/software/jira