[jira] [Commented] (YARN-4108) CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request

["Bikas Saha (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-4108?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15028104#comment-15028104 ] 

Bikas Saha commented on YARN-4108:
----------------------------------

These problems will be hard to solve without involving the scheduler in the decision cycle. The preemption policy can determine how much to preempt from a queue at a macro level. But the actual containers to preempt would be selected by the scheduler.
That is where using the global node picture will help. For a given container request, if we can scan its nodes (if any) and make  either an allocation or preemption decision. 
Else, if we are doing container allocation on node heartbeat, then just like delay scheduling logic, we can mark a node for preemption but not preempt it and associate that node with the container request for which preemption is needed (request.nodeToPreempt). And we can cycle through all nodes like this and change the request->node association when we find better nodes to preempt. After cycling through all nodes, if when we again reach a node such that it matches the request.nodeToPreempt then we can execute the decision of actually preempting the node. If there are no nodes that can satisfy the request (e.g. request wants node A but preemptedQueue has no containers on node A) then scheduler should be able to callback to the preemption module and notify it so that some other queue can be picked to preempt.

> CapacityScheduler: Improve preemption to preempt only those containers that would satisfy the incoming request
> --------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-4108
>                 URL: https://issues.apache.org/jira/browse/YARN-4108
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Assignee: Wangda Tan
>
> This is sibling JIRA for YARN-2154. We should make sure container preemption is more effective.
> *Requirements:*:
> 1) Can handle case of user-limit preemption
> 2) Can handle case of resource placement requirements, such as: hard-locality (I only want to use rack-1) / node-constraints (YARN-3409) / black-list (I don't want to use rack1 and host\[1-3\])
> 3) Can handle preemption within a queue: cross user preemption (YARN-2113), cross applicaiton preemption (such as priority-based (YARN-1963) / fairness-based (YARN-3319)).



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)