[jira] [Commented] (YARN-1961) Fair scheduler preemption doesn't work for non-leaf queues

["Karthik Kambatla (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-1961?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15186273#comment-15186273 ] 

Karthik Kambatla commented on YARN-1961:
----------------------------------------

Configuring minResources and minshare preemption on non-leaf queues might not be desirable for the following reasons:
# Minshare preemption was often used in the past to address cases where users wanted to be able to trigger preemption when a queue is over half of its fairshare. Now that we have added fairsharePreemptionThreshold, this is less the case.
# Other usecase for minshare preemption was to give enough resources for a high priority job quickly (shorter minshare preemption timeout). Now, we support setting the fairsharePreemptionTimeout per queue. Also, while it might be easier to figure out resources need for an application, it is much harder to come up with absolute minimum resources required for any queue in general and a non-leaf queue in particular. 

[~ashwinshankar77] - if you agree this is less desirable, I would like to close this as "Won't Fix". 

> Fair scheduler preemption doesn't work for non-leaf queues
> ----------------------------------------------------------
>
>                 Key: YARN-1961
>                 URL: https://issues.apache.org/jira/browse/YARN-1961
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: fairscheduler, scheduler
>    Affects Versions: 2.4.0
>            Reporter: Ashwin Shankar
>            Assignee: Ashwin Shankar
>              Labels: scheduler
>
> Setting minResources and minSharePreemptionTimeout to a non-leaf queue doesn't cause preemption to happen when that non-leaf queue is below minResources and there are outstanding demands in that non-leaf queue.
> Here is an example fs allocation config(partial) :
> {code:xml}
> <queue name="abc">
>   <minResources>3072 mb,0 vcores</minResources>
>   <minSharePreemptionTimeout>30</minSharePreemptionTimeout>
>     <queue name="childabc1">
>     </queue>
>     <queue name="childabc2">
>     </queue>
>  </queue>
>  {code}
> With the above configs,preemption doesn't seem to happen if queue abc is below minShare and it has outstanding unsatisfied demands from apps in its child queues. Ideally in such cases we would like preemption to kick off and reclaim resources from other queues(not under queue abc).
> Looking at the code it seems like preemption checks for starvation only at the leaf queue level and not at the parent level.
> {code:title=FairScheduler.java|borderStyle=solid}
> boolean isStarvedForMinShare(FSLeafQueue sched)
> boolean isStarvedForFairShare(FSLeafQueue sched)
> {code}
> This affects our use case where we have a parent queue with probably a 100 unconfigured leaf queues under it.We want to give a minshare to the parent queue to protect all the leaf queues under it,but we cannot do it due to this bug.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)