[jira] [Commented] (YARN-10559) Fair sharing intra-queue preemption support in Capacity Scheduler

["Eric Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10559?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17294871#comment-17294871 ] 

Eric Payne commented on YARN-10559:
-----------------------------------

[~ananyo_rao], sorry for the delay. I'm trying to get my head around what I think the proper solution should be for this problem. I think that the crux of it is that in {{FifoIntraQueuePreemptionPlugin#validateOutSameAppPriorityFromDemand}}, it does not allow preemption if from the same user, and in {{FifoIntraQueuePreemptionPlugin#skipContainerBasedOnIntraQueuePolicy}} it doesn't allow the user to get below it's user limit.

In this case, we don't care if the preemption will cause the user to go below it's user limit because we expect the container to go back to the same user, just in a different app. However, since the state of the queue and cluster is always in flux, there is no guarantee that the preempted container will go to the app we expect it to.

Simply skipping these 2 checks is not sufficient either, since that will cause over-preemption to happen, with containers being preempted and then being assigned back to the same app it preempted them from.

> Fair sharing intra-queue preemption support in Capacity Scheduler
> -----------------------------------------------------------------
>
>                 Key: YARN-10559
>                 URL: https://issues.apache.org/jira/browse/YARN-10559
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 3.1.4
>            Reporter: VADAGA ANANYO RAO
>            Assignee: VADAGA ANANYO RAO
>            Priority: Major
>         Attachments: FairOP_preemption-design_doc_v1.pdf, FairOP_preemption-design_doc_v2.pdf, YARN-10559.0001.patch, YARN-10559.0002.patch, YARN-10559.0003.patch, YARN-10559.0004.patch, YARN-10559.0005.patch, YARN-10559.0006.patch, YARN-10559.0007.patch, YARN-10559.0008.patch, YARN-10559.0009.patch
>
>   Original Estimate: 168h
>  Remaining Estimate: 168h
>
> Usecase:
> Due to the way Capacity Scheduler preemption works, If a single user submits a large application to a queue (using 100% of resources), that job will not be preempted by future applications from the same user within the same queue. This implies that the later applications will be forced to wait for completion of the long running application. This prevents multiple long running, large, applications from running concurrently.
> Support fair sharing among apps while preempting applications from same queue.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org