[jira] [Commented] (YARN-373) Allow an AM to reuse the resources allocated to container for a new container

["Hitesh Shah (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-373?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13569659#comment-13569659 ] 

Hitesh Shah commented on YARN-373:
----------------------------------

@Alejandro, had a few questions regarding the jira description:
  
  - is this only for re-sizing an existing allocated container in terms of the cpu/memory resources allocated to it? i.e either increase or decrease the size of the container ( reduce memory as you mentioned for shuffle or increase it for allocating a reduce task on it for example ) ? 
  - the container today exits as soon as the process completes as part of the container. Is the proposal also to change the MR tasks today to convert and launch a shuffle handler within the same process space after the map task completes? Or change the current notion of what constitutes completion of a container? 
  - would the new 'task' running on the container need to run with the same existing set of localized resources or would there be new localization needed?



 
                
> Allow an AM to reuse the resources allocated to container for a new container
> -----------------------------------------------------------------------------
>
>                 Key: YARN-373
>                 URL: https://issues.apache.org/jira/browse/YARN-373
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: resourcemanager
>    Affects Versions: 2.0.3-alpha
>            Reporter: Alejandro Abdelnur
>            Assignee: Alejandro Abdelnur
>
> When a container completes, instead the corresponding resources being freed up, it should be possible for the AM to reuse the assigned resources for a new container.
> As part of the reallocation, the AM would notify the RM about partial resources being freed up and the RM would make the necessary corrections in the corresponding node.
> With this functionality, an AM can ensure it gets a container in the same node where previous containers run.
> This will allow getting rid of the ShuffleHandler as a service in the NMs and run it as regular container task of the corresponding AM. In this case, the reallocation would reduce the CPU/MEM obtained for the original container to the what is needed for serving the shuffle. Note that in this example the MR AM would only do this reallocation for one of the many tasks that may have run in a particular node (as a single shuffle task could serve all the map outputs from all map tasks run in that node). 

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira