[jira] [Updated] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request

["Wangda Tan (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Wangda Tan updated YARN-4108:
-----------------------------
    Attachment: YARN-4108-design-doc-V3.pdf

Uploaded a new doc/poc patch, the proposal of this doc is more conservative, highlights of the doc:

*Steps*
- Leave existing PCPP as-is, it computes ideal share of each queue, and select to-be-preempt containers like before.
- When a container expires kill-wait (could be killed), “kill container” message will be sent to scheduler, and scheduler marks container to be “killable”
- When scheduler trying to allocate an under-satisfied queue, it will try to deduct resources of “killable” containers. If a new container could be allocated/reserved after killing these “killable” containers. Scheduler will pull the trigger immediately.

In short, this proposal avoids awkwardness that preempted resources cannot be used by who needs the resource, and then such preempted resources come back to over-utilized queue again.

More details please refer to doc/WIP-patch. Please feel free to let me know if you have any comments.

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-4108-design-doc-V3.pdf, YARN-4108-design-doc-v1.pdf, YARN-4108-design-doc-v2.pdf, YARN-4108.poc.1.patch
>
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)