[jira] [Commented] (YARN-1902) Allocation of too many containers when a second request is done with the same resource capability

["Sietse T. Au (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1902?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14212100#comment-14212100 ] 

Sietse T. Au commented on YARN-1902:
------------------------------------

The scenarios are such that ContainerRequest is a separate object with each addContainerRequest call.

The point is that the user does not want its previous requests to be undone, but to do an additional request, while not yet having received all the 12 containers. 

Let's say for Scenario 1 that z = 12
12 containers are requested, then asynchronously you start an allocated container, after this, 1 extra container is requested. However instead of 1 extra container being requested, 13 containers will be requested. This - as far as I can see - does not necessarily involve the RM and could be solved at the client side's internal bookkeeping of requests, which is what the patch is trying to show.

> Allocation of too many containers when a second request is done with the same resource capability
> -------------------------------------------------------------------------------------------------
>
>                 Key: YARN-1902
>                 URL: https://issues.apache.org/jira/browse/YARN-1902
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: client
>    Affects Versions: 2.2.0, 2.3.0, 2.4.0
>            Reporter: Sietse T. Au
>              Labels: client
>         Attachments: YARN-1902.patch, YARN-1902.v2.patch, YARN-1902.v3.patch
>
>
> Regarding AMRMClientImpl
> Scenario 1:
> Given a ContainerRequest x with Resource y, when addContainerRequest is called z times with x, allocate is called and at least one of the z allocated containers is started, then if another addContainerRequest call is done and subsequently an allocate call to the RM, (z+1) containers will be allocated, where 1 container is expected.
> Scenario 2:
> No containers are started between the allocate calls. 
> Analyzing debug logs of the AMRMClientImpl, I have found that indeed a (z+1) are requested in both scenarios, but that only in the second scenario, the correct behavior is observed.
> Looking at the implementation I have found that this (z+1) request is caused by the structure of the remoteRequestsTable. The consequence of Map<Resource, ResourceRequestInfo> is that ResourceRequestInfo does not hold any information about whether a request has been sent to the RM yet or not.
> There are workarounds for this, such as releasing the excess containers received.
> The solution implemented is to initialize a new ResourceRequest in ResourceRequestInfo when a request has been successfully sent to the RM.
> The patch includes a test in which scenario one is tested.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)