[jira] [Comment Edited] (YARN-5501) Container Pooling in YARN

["Hitesh Sharma (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15414834#comment-15414834 ] 

Hitesh Sharma edited comment on YARN-5501 at 8/10/16 7:11 AM:
--------------------------------------------------------------

Thanks [~atris] for the comments. These are good points but the answers depend upon the implementation of the container. 

1) Yes, there will be some overhead of maintaining the pooled containers but that's a trade off to optimize for launch latencies. Containers can however implement some custom behaviors to lower the overhead. As an e.g. if the container supports PAUSE and RESUME semantics ([YARN-5292]) then a pooled container could be PAUSED. Some other container could however chose to resize the allocation to a minimum and resize as per the actual resource request.

2) I'm not sure if I follow the comment here. Pooled containers are useful to lower the launch latencies and that's independent of the actual container run time.

3) That would be implementation specific. A pooled container is in itself a resource and when acquired by a request would need to be adjusted accordingly.

We will be posting some more design and implementation details which will hopefully help clarify the ideas here.


was (Author: hrsharma):
Thanks [~atris] for the comments. These are good points but the answers depend upon the implementation of the container. 

1) Yes, there will be some overhead of maintaining the pooled containers but that's a trade off to optimize for launch latencies. Containers can however implement some custom behaviors to lower the overhead. As an e.g. if the container supports PAUSE and RESUME semantics [YARN-5292] then a pooled container could be PAUSED. Some other container could however chose to resize the allocation to a minimum and resize as per the actual resource request.

2) I'm not sure if I follow the comment here. Pooled containers are useful to lower the launch latencies and that's independent of the actual container run time.

3) That would be implementation specific. A pooled container is in itself a resource and when acquired by a request would need to be adjusted accordingly.

We will be posting some more design and implementation details which will hopefully help clarify the ideas here.

> Container Pooling in YARN
> -------------------------
>
>                 Key: YARN-5501
>                 URL: https://issues.apache.org/jira/browse/YARN-5501
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Arun Suresh
>
> This JIRA proposes a method for reducing the container launch latency in YARN. It introduces a notion of pooling *Unattached Pre-Initialized Containers*.
> Proposal in brief:
> * Have a *Pre-Initialized Container Factory* service within the NM to create these unattached containers.
> * The NM would then advertise these containers as special resource types (this should be possible via YARN-3926).
> * When a start container request is received by the node manager for launching a container requesting this specific type of resource, it will take one of these unattached pre-initialized containers from the pool, and use it to service the container request.
> * Once the request is complete, the pre-initialized container would be released and ready to serve another request.
> This capability would help reduce container launch latencies and thereby allow for development of more interactive applications on YARN.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org