[jira] [Commented] (YARN-6706) Refactor ContainerScheduler to make oversubscription change easier

["Haibo Chen (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6706?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16045229#comment-16045229 ] 

Haibo Chen commented on YARN-6706:
----------------------------------

This patch does the following:
1) Changes the ContainerScheduler.scheduleContainer() logic
if the container is guaranteed, we first put it into the guaranteed container queue, then do a pass-through guaranteed queue followed by opportunistic queue to start as many containers as possible. Finally, if the container remains in the guaranteed queue, we know we need to kill some opportunistic container

if the container is opportunistic, we first do a pass through pass-through guaranteed queue followed by opportunistic queue to start as many containers as possible. Then we try to enqueue the opportunistic container, which may fail if the maxOppQueue length is reached. Lastly, we may be able to start it if there happen to be some resources remaining. This way, we slightly change the semantic of maxOppQueueLength in cases where oversubscription is enabled. We enqueue an opportunistic container when neither there is unallocated resource nor can it be started due to oversubscription.

2)  When maxOppQUeueLength is set to 0 or a negative value, containers are no longer unconditionally started. 

3) Adds a new unit test in TestContainerSchedulerQueuing to test the case where the OppQueue length limit is reached with too many opportunistic container requests.

4) Add a new method resourceAvailableToStartContainer() which we could extend to include oversubscription logic.

> Refactor ContainerScheduler to make oversubscription change easier
> ------------------------------------------------------------------
>
>                 Key: YARN-6706
>                 URL: https://issues.apache.org/jira/browse/YARN-6706
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>    Affects Versions: 3.0.0-alpha3
>            Reporter: Haibo Chen
>            Assignee: Haibo Chen
>         Attachments: YARN-6706-YARN-1011.00.patch
>
>




--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org