You are viewing a plain text version of this content. The canonical link for it is here.
Posted to reviews@yunikorn.apache.org by wi...@apache.org on 2022/12/19 04:59:48 UTC
[yunikorn-site] branch master updated: [YUNIKORN-1475] Document priority support (#232)
This is an automated email from the ASF dual-hosted git repository.
wilfreds pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 619042d6e [YUNIKORN-1475] Document priority support (#232)
619042d6e is described below
commit 619042d6e8c51f5917fd727c8c5981e65600ff1d
Author: Craig Condit <cc...@apache.org>
AuthorDate: Mon Dec 19 15:59:32 2022 +1100
[YUNIKORN-1475] Document priority support (#232)
Closes: #232
Signed-off-by: Wilfred Spiegelenburg <wi...@apache.org>
---
docs/assets/priority-tree.png | Bin 0 -> 142135 bytes
docs/user_guide/priorities.md | 220 ++++++++++++++++++++++++++++++++++++
docs/user_guide/queue_config.md | 89 +++++++++++++--
docs/user_guide/sorting_policies.md | 30 +++--
sidebars.js | 1 +
5 files changed, 322 insertions(+), 18 deletions(-)
diff --git a/docs/assets/priority-tree.png b/docs/assets/priority-tree.png
new file mode 100644
index 000000000..7288db5ea
Binary files /dev/null and b/docs/assets/priority-tree.png differ
diff --git a/docs/user_guide/priorities.md b/docs/user_guide/priorities.md
new file mode 100644
index 000000000..00b5836b5
--- /dev/null
+++ b/docs/user_guide/priorities.md
@@ -0,0 +1,220 @@
+---
+id: priorities
+title: App & Queue Priorities
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+YuniKorn has advanced support for priority scheduling. Priorities are
+specified on a per-task basis, but how those priorities are used can be
+customized for each queue.
+
+## Request Priority
+
+Every allocation request to the scheduler has a numeric priority associated
+with it. Any 32-bit integer value (positive or negative) may be used. Larger
+values indicate higher relative priorities.
+
+When using Kubernetes, priorities are defined in `PriorityClass`
+objects, which are referenced by `Pod` objects via a `priorityClassName`
+property. If no priority class is referenced, a `Pod` inherits the cluster
+default priority, typically `0`.
+
+See the Kubernetes [Pod Priority and Preemption](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/)
+documentation for more details.
+
+## Application Priority
+
+During scheduling, applications have a dynamic priority value which resolves
+to the highest priority outstanding request in that application. This allows
+the scheduler to dynamically reprioritize scheduling decisions.
+
+For example, if an application has two requests, one with priority `10` and
+another with priority `20`, the application's dynamic priority will be `20`.
+If the higher-priority request is satisfied, the application's priority will
+drop to `10`.
+
+When choosing between applications to schedule, the application sorting policy
+will (by default) schedule requests from higher-priority applications first.
+Priority can be ignored when sorting applications by setting the queue
+property `application.sort.priority` to `disabled` on a _leaf_ queue.
+
+## Queue Priority
+
+As with applications, queue priorities are also dynamically computed. For
+_parent_ queues, the queue's priority will be equal to the highest priority
+child queue it contains. For _leaf_ queues, the queue's priority will be
+equal to the highest priority application it contains.
+
+Queue priorities can also be adjusted automatically up or down by a fixed
+amount, specified in the `priority.offset` queue property. This can be useful
+in larger queue hierarchies to establish low or high priority queues.
+
+For example, if a _leaf_ queue with an offset of `5` contains two
+applications, one with priority `10` and another with priority `20`, the
+queue's priority will evaluate to `25` (`20 + 5`). If the higher-priority
+application no longer has requests, the queue's priority will drop to `15`
+(`10 + 5`).
+
+When choosing between child queues to schedule from, the queue sorting policy
+will (by default) schedule requests from higher-priority queues first.
+Priority can be ignored when sorting queues by setting the queue
+property `application.sort.priority` to `disabled` on a _parent_ queue.
+
+## Priority Fencing
+
+By default, priorities have global scope, meaning that higher-priority queues
+will be serviced before lower-priority queues regardless of their location
+in the queue hierarchy.
+
+However, it can be useful to limit prioritization to a subset of queues. This
+can be accomplished by setting the `priority.policy` queue property to
+`fence`. When a queue's priority is fenced, priorities are still evaluated
+within that queue (and subqueues), but the queue's priority itself will always
+evaluate to the `priority.offset` value or `0` if not specified.
+
+## Effective Priority
+
+Because of offsets and fencing, at any time a request may be thought of as
+having an _effective_ (or computed) priority based on its location within
+the queue hierarchy. Requests with higher effective priorities will be
+scheduled before those with lower effective priorities.
+
+## Examples
+
+### Single queue
+
+This example demonstrates a single leaf queue with all properties specified:
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: default
+ properties:
+ application.sort.policy: fifo
+ application.sort.priority: enabled
+ priority.policy: default
+ priority.offset: "0"
+```
+
+### Multitenancy
+
+This example demonstrates a complex queue tree containing multiple tenants
+with subqueues along with a multiple system queues:
+
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: system
+ queues:
+ - name: system-normal
+ properties:
+ priority.offset: "0"
+ - name: system-high
+ properties:
+ priority.offset: "1000"
+ - name: system-low
+ properties:
+ priority.offset: "-1000"
+ - name: tenants
+ properties:
+ priority.policy: "fence"
+ queues:
+ - name: tenant-a
+ properties:
+ priority.policy: "fence"
+ priority.offset: "10"
+ queues:
+ - name: child-a-1
+ - name: child-a-2
+ - name: tenant-b
+ properties:
+ priority.policy: "fence"
+ priority.offset: "0"
+ queues:
+ - name: child-b-1
+ - name: child-b-2
+
+```
+
+
+The `system-normal`, `system-high` and `system-low` queues are unfenced, and
+can therefore be prioritized above any other queue in the system. The
+`system-high` and `system-low` queues have offsets of `1000` and `-1000`
+respectively, so the priority of requests in those queues will be adjusted
+accordingly.
+
+The `tenants` _parent_ queue is priority-fenced, and has no `priority.offset`
+defined, so this queue will always be treated as though it has priority `0`.
+This ensures that normal and high-priority system tasks schedule ahead of
+tenant tasks, and low priority system tasks schedule after tenant tasks.
+
+The `tenant-a` and `tenant-b` _parent_ queues are also priority-fenced,
+preventing tenants from adjusting their priority relative to one another.
+The `tenant-a` queue also has a priority offset to ensure that it always
+schedules before `tenant-b`.
+
+The _leaf_ queues of `tenant-a` and `tenant-b` are not fenced, so tasks from
+the entire `tenant-a` or `tenant-b` subtree will prioritize relative to each
+other, but not outside their respective subtrees.
+
+
+
+In the figure above, multiple requests are shown with various priorities.
+Before scheduling, the queue priorities will be as follows:
+
+* root
+ * system: 1001
+ * system-normal: 10
+ * system-high: 1001
+ * system-low: -997
+ * tenants: 0 (fence)
+ * tenant-a: 10 (fence)
+ * child-a-1: 8
+ * child-a-2: 6
+ * tenant-b: 0 (fence)
+ * child-b-1: 9
+ * child-b-2: 8
+
+Queue traversal starts from the root, descending into each child queue in order
+starting with the highest effective priority. Assuming sufficient scheduling
+resources, the order of schedulding and effective queue priority changes are
+as follows:
+
+| Step | Queue | Task | Result |
+|------|---------------------------------|------|----------------------------------------------------------------------------------------|
+| 1 | root.system.system-high | P1 | **system-high**: `1001` -> n/a <br/> **system**: `1001` -> `10` |
+| 2 | root.system.system-normal | P10 | **system-normal**: `10` -> `2` <br/> **system**: `10` -> `2` |
+| 3 | root.system.system-normal | P2 | **system-normal**: `2` -> n/a <br/> **system**: `2` -> `-997` |
+| 4 | root.tenants.tenant-a.child-a-1 | P8 | **child-a-1**: `8` -> `5` |
+| 5 | root.tenants.tenant-a.child-a-2 | P6 | **child-a-2**: `6` -> `4` |
+| 6 | root.tenants.tenant-a.child-a-1 | P5 | **child-a-1**: `5` -> n/a |
+| 7 | root.tenants.tenant-a.child-a-2 | P4 | **child-a-2**: `4` -> n/a <br/> **tenant-a**: `10` -> n/a |
+| 8 | root.tenants.tenant-b.child-b-1 | P9 | **child-b-1**: `9` -> `7` |
+| 9 | root.tenants.tenant-b.child-b-2 | P8 | **child-b-2**: `8` -> n/a |
+| 10 | root.tenants.tenant-b.child-b-1 | P7 | **child-b-1**: `7` -> n/a <br/> **tenant-b**: `0` -> n/a <br/> **tenants**: `0` -> n/a |
+| 11 | root.system.system-low | P3 | **system-low**: `-997` -> n/a <br/> **system**: `-997` -> n/a |
+
diff --git a/docs/user_guide/queue_config.md b/docs/user_guide/queue_config.md
index a128c6004..31a0005f2 100644
--- a/docs/user_guide/queue_config.md
+++ b/docs/user_guide/queue_config.md
@@ -133,7 +133,7 @@ Supported parameters for the queues:
* parent
* queues
* maxapplications
-* properties
+* [properties](#properties)
* adminacl
* submitacl
* [resources](#resources)
@@ -159,12 +159,13 @@ Trying to override a _parent_ queue type in the configuration will cause a parsi
Sub queues for a parent queue are defined under the `queues` entry.
The `queues` entry is a recursive entry for a queue level and uses the exact same set of parameters.
-The _maxapplications_ property is an integer value, larger than 1, which allows you to limit the number of running applications for the queue. Specifying a zero for _maxapplications_ is not allowed as it would block all allocations for applications in the queue. The _maxapplications_ value for a _child_ queue must be smaller or equal to the value for the _parent_ queue.
-The `properties` parameter is a simple key value pair list.
-The list provides a simple set of properties for the queue.
-There are no limitations on the key or value values, anything is allowed.
-Currently, the property list is only used in the scheduler to define a [sorting order](sorting_policies.md#application-sorting) for a leaf queue.
-Future expansions, like the option to turn on or off preemption on a queue or other sorting policies, would use this same property construct without the need to change the configuration.
+The `maxapplications` property is an integer value, larger than 1, which allows you to limit the number of running applications for the queue. Specifying a zero for `maxapplications` is not allowed as it would block all allocations for applications in the queue. The `maxapplications` value for a _child_ queue must be smaller or equal to the value for the _parent_ queue.
+
+The [properties](#properties) section contains simple key/value pairs. This is
+used for further queue customization of features such as
+[application sorting](sorting_policies.md#application-sorting) and priority
+scheduling. Future features will use the exisitng `properties` section as well
+to avoid the need to define a new structure for queue configuration.
Access to a queue is set via the `adminacl` for administrative actions and for submitting an application via the `submitacl` entry.
ACLs are documented in the [Access control lists](acls.md) document.
@@ -299,6 +300,80 @@ users:
```
In this case both the users `sue` and `bob` are allowed to run 10 applications.
+### Properties
+
+Additional queue configuration can be added via the `properties` section,
+specified as simple key/value pairs. The following parameters are currently
+supported:
+
+#### `application.sort.policy`
+
+Supported values: `fifo`, `fair`, `stateaware`
+
+Default value: `fifo`
+
+Sets the policy to be used when sorting applications within a queue. This
+setting has no effect on a _parent_ queue.
+
+See the documentation on [application sorting](sorting_policies.md#application-sorting)
+for more information.
+
+
+#### `application.sort.priority`
+
+Supported values: `enabled`, `disabled`
+
+Default value: `enabled`
+
+When this property is `enabled`, priority will be considered when sorting
+queues and applications. Setting this value to `disabled` will ignore
+priorities when sorting. This setting can be specified on a _parent_ queue and
+will be inherited by _child_ queues.
+
+**NOTE:** YuniKorn releases prior to 1.2.0 did not support priorities when
+sorting. To keep the legacy behavior, set `application.sort.priority` to
+`disabled`.
+
+#### `priority.policy`
+
+Supported values: `default`, `fence`
+
+Default value: `default`
+
+Sets the inter-queue priority policy to use when scheduling requests.
+
+**NOTE**: This value is not inherited by child queues.
+
+By default, priority applies across queues globally. In other words,
+higher-priority requests will be satisfied prior to lower-priority requests
+regardless of which queue they exist within.
+
+When the `fence` policy is in use on a queue, the priorities of _child_ queues
+(in the case of a _parent_ queue) or applications (in the case of a _leaf_
+queue) will not be exposed outside the fence boundary.
+
+See the documentation on [priority support](priorities.md) for more information.
+
+#### `priority.offset`
+
+Supported values: any positive or negative 32-bit integer
+
+Default value: `0`
+
+Adjusts the priority of the queue relative to it's siblings. This can be useful
+to create high or low-priority queues without needing to set every task's
+priority manually.
+
+**NOTE**: This value is not inherited by child queues.
+
+When using the `default` priority policy, the queue's priority is adjusted up
+or down by this amount.
+
+When using the `fence` policy, the queue's priority is always set to the offset
+value (in other words, the priorities of tasks in the queue are ignored).
+
+See the documentation on [priority support](priorities.md) for more information.
+
### Resources
The resources entry for the queue can set the _guaranteed_ and or _maximum_ resources for a queue.
Resource limits are checked recursively.
diff --git a/docs/user_guide/sorting_policies.md b/docs/user_guide/sorting_policies.md
index c5f4bfcff..413d1f1c2 100644
--- a/docs/user_guide/sorting_policies.md
+++ b/docs/user_guide/sorting_policies.md
@@ -29,8 +29,8 @@ Policies can be set for:
## Application sorting
The application sorting policy is set for each queue via the config.
-A sorting policy setting is only effective on a `leaf` queue.
-Each `leaf` queue can use a different policy.
+A sorting policy setting is only effective on a _leaf_ queue.
+Each _leaf_ queue can use a different policy.
A sorting policy only specifies the order in which the applications are sorted within a queue.
That order is crucial in specifying which application is considered first when assigning resources.
@@ -38,7 +38,12 @@ Sorting policies do _not_ affect the number of applications that are scheduled o
All applications that have pending resource requests can and will be scheduled in a queue unless specifically filtered out.
Even when applications are sorted using a first in first out policy multiple applications will run in a queue in parallel.
-A `parent` queue will always use the fair policy to sort the child queues.
+A _parent_ queue will always use the fair policy to sort the child queues.
+
+The relative priority of child queues (in the case of _parent_ queue sorting)
+and applciations (in the case of _leaf_ queue sorting) will be considered first.
+To ignore application and queue priorities when scheduling, set the queue
+property `application.sort.priority` to `disabled`.
The following configuration entry sets the application sorting policy to `fifo` for the queue `root.sandbox`:
```yaml
@@ -57,9 +62,10 @@ A filter is applied _while_ sorting the applications to remove all that do not h
### FifoSortPolicy
Short description: first in first out, based on application create time
-Config value: fifo (default)
-Behaviour:
-Before sorting the applications are filtered and must have pending resource requests.
+
+Config value: `fifo` (default)
+
+Before sorting, the applications are filtered and must have pending resource requests.
After filtering the applications left are sorted based on the application create time stamp only, no other filtering is applied.
Since applications can only be added while the system is locked there can never be two applications with the exact same time stamp.
@@ -69,8 +75,9 @@ Younger applications will be given resources when all the current requests of ol
### FairSortPolicy
Short description: fair based on usage
-Config value: fair
-Behaviour:
+
+Config value: `fair`
+
Before sorting the applications are filtered and must have pending resource requests.
After filtering the applications left are sorted based on the application usage.
@@ -81,8 +88,9 @@ The result is that the resources available are spread equally over all applicati
### StateAwarePolicy
Short description: limit of one (1) application in Starting or Accepted state
-Config value: stateaware
-Behaviour:
+
+Config value: `stateaware`
+
This sorting policy requires an understanding of the application states.
Applications states are described in the [application states](design/scheduler_object_states.md#application-state) documentation.
@@ -182,4 +190,4 @@ There is currently one policy for sorting requests within an application.
This policy is not configurable.
Sorting requests is only possible based on the priority of the request.
If there are multiple requests within an application that have the same priority the order of the requests is undetermined.
-This means that the order of requests with the same priority can, and most likely will, change between runs.
\ No newline at end of file
+This means that the order of requests with the same priority can, and most likely will, change between runs.
diff --git a/sidebars.js b/sidebars.js
index f3b0e1963..e68867f9e 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -27,6 +27,7 @@ module.exports = {
'user_guide/placement_rules',
'user_guide/usergroup_resolution',
'user_guide/sorting_policies',
+ 'user_guide/priorities',
'user_guide/acls',
'user_guide/resource_quota_management',
'user_guide/gang_scheduling',