You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by "Tao Yang (Jira)" <ji...@apache.org> on 2021/01/07 02:37:00 UTC
[jira] [Commented] (YUNIKORN-1) Support app/task priority aware
scheduling
[ https://issues.apache.org/jira/browse/YUNIKORN-1?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17260177#comment-17260177 ]
Tao Yang commented on YUNIKORN-1:
---------------------------------
Hi, [~wangda] [~wwei] [~wilfreds]
As the refactor for core scheduling process has been done, I would like to keep this ticket moving. This time I think we can establish a pluggable mechanism with flexible interfaces at first.
Currently the key steps of scheduling process such as applications/requests sorting are implemented with fixed process and policies, users may have some different requirements on them but it's not flexible enough. Moreover, some steps are relatively time-consuming in every scheduling cycle, we can improve the performance a lot via different implementations if they are pluggable.
I have tried to improve following places:
(1) requests inside the application which are managed via a map structure.
To make it pluggable, at first we should add a Request interface out of scheduler/objects package for AllocationAsk to avoid circular dependencies, then add Requests interface with some common methods like AddRequest/RemoveRequest/GetRequest in interfaces package, and add a default implemetation DefaultRequests with map structure for it, at last add RequestsPlugin interface and its default implementation in plugins package which helps to generate new requests instance when creating an application.
(2) key steps about sorting applications and requests inside a queue.
Currently sorting applications and sorting requests are separated so that some requirements can't be satisfied, for example we have 3 apps APP1/APP2, every application have different priorities(APP1: 3/5, APP2: 2/4), the request priority is utmost important so that the required sequence may be APP1/priority=5 > APP2/priority=4 > APP1/priority=3 > APP2/priority=2. Thus I propose to add QueueRequestManager interface to handle these steps together to make it more flexible, it may have simple method like SortForAllocation() whose return is an iterator which can output an application and its requests, AppRequestIterator/RequestIterator interfaces are added for iterating applications and requests, QueueRequestManagerPlugin interface is imported as RequestsPlugin which helps to generate QueueRequestManager instance when initilizing a queue.
Please refer to the [draft PR|https://github.com/apache/incubator-yunikorn-core/pull/237] for details. Hope to hear your thoughts, Thanks.
> Support app/task priority aware scheduling
> ------------------------------------------
>
> Key: YUNIKORN-1
> URL: https://issues.apache.org/jira/browse/YUNIKORN-1
> Project: Apache YuniKorn
> Issue Type: New Feature
> Components: core - scheduler
> Reporter: Weiwei Yang
> Assignee: Tao Yang
> Priority: Major
> Labels: pull-request-available
> Attachments: Support priority-based scheduling among applications v1.pdf, Support priority-based scheduling among applications v2.pdf, Support priority-based scheduling among applications v3.pdf
>
> Time Spent: 10m
> Remaining Estimate: 0h
>
> The scheduler needs to understand app/task priority and make proper decisions while allocating resources. I think we don't need to introduce a very complex model when defining priorities, e.g supporting both queues, app, and task level priorities, it will be hard for users to consume. Instead, we need to find a clean, straightforward approach to handle priorities.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org