[jira] [Commented] (YARN-6151) FS Preemption doesn't filter out queues which cannot be preempted

["Yufei Gu (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6151?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15854730#comment-15854730 ] 

Yufei Gu commented on YARN-6151:
--------------------------------

Uploaded one simple solution is that before we choose which queue to preempt, we check if this queue can be preempted. For leaf queue, we just check if the queue uses more resources than its fair share. For parent queue, we need to recursively go down to each level to check. There might be a performance concern, but considering the performance is still acceptable before YARN-5969 which did recursively get resource usage anyway, it should be fine.

> FS Preemption doesn't filter out queues which cannot be preempted
> -----------------------------------------------------------------
>
>                 Key: YARN-6151
>                 URL: https://issues.apache.org/jira/browse/YARN-6151
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: fairscheduler
>    Affects Versions: 2.8.0
>            Reporter: Yufei Gu
>            Assignee: Yufei Gu
>         Attachments: YARN-6151.001.patch
>
>
> This is preemption bug happens before 2.8.0, which also described in YARN-3405.
> Queue hierarchy described as below:
> {noformat}
>                   root
>                /         \
>        queue-1          queue-2       
>           /      \
> queue-1-1     queue-1-2
> {noformat}
> Assume cluster resource is 100 and all queues have same weights.
> # queue-1-1 and queue-2 has apps. Each get 50 usage and 50 fairshare. 
> # When queue-1-2 is active, supposedly it will preempt 25 from queue-1-1, but this doesn't happen because preemption happens top-down, queue-2 could be the preemption candidate as long as queue-2 is less needy than queue-1, and queue-2 doesn't exceed the fair share which means preemption won't happen.
> We need to filter out queue-2 since it isn't a valid candidate.



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org