[jira] [Commented] (YARN-5864) Capacity Scheduler preemption for fragmented cluster

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-5864?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15752695#comment-15752695 ] 

Wangda Tan commented on YARN-5864:
----------------------------------

Offline discussed with [~vinodkv].

We can have a better semantic of this feature, which is we can add queue-priority property. (Credit to [~vinodkv] for the idea).

In existing scheduler, we sort queues based on (used-capacity / configured-capacity). But in some cases we have some apps/services need get resource first. For example, we allocate 85% to production queue, and 15% to test queue. When production queue is underutilized, we want scheduler give resource to production queue first regardless of test queue's utilization.

A rough plan is: we will assign priority to queues under the same parent. Each time scheduler picks underutilized queue with highest priority, if there's no underutilized queue, scheduler picks queue with lowest utilization.

And when we do preemption, if queue with higher priority has some special resource requests, such as very large memory, hard locality, placement constraint, etc. Scheduler will do relatively *conservative* preemption from other queues with lower priority regardless of utilization. 

That is just a rough idea, [~curino] please let us know your comments. I can formalize the design once we can agree with the approach generally.

> Capacity Scheduler preemption for fragmented cluster 
> -----------------------------------------------------
>
>                 Key: YARN-5864
>                 URL: https://issues.apache.org/jira/browse/YARN-5864
>             Project: Hadoop YARN
>          Issue Type: New Feature
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>         Attachments: YARN-5864.poc-0.patch
>
>
> YARN-4390 added preemption for reserved container. However, we found one case that large container cannot be allocated even if all queues are under their limit.
> For example, we have:
> {code}
> Two queues, a and b, capacity 50:50 
> Two nodes: n1 and n2, each of them have 50 resource 
> Now queue-a uses 10 on n1 and 10 on n2
> queue-b asks for one single container with resource=45. 
> {code} 
> The container could be reserved on any of the host, but no preemption will happen because all queues are under their limits. 



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org