You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by ma...@apache.org on 2026/01/22 08:09:35 UTC
(yunikorn-site) branch master updated: [YUNIKORN-3134] Publish design doc (#538)
This is an automated email from the ASF dual-hosted git repository.
mani pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 7d583cfcde [YUNIKORN-3134] Publish design doc (#538)
7d583cfcde is described below
commit 7d583cfcded320a6a8edf3f284c5e04e53338f50
Author: Manikandan R <ma...@gmail.com>
AuthorDate: Thu Jan 22 13:39:20 2026 +0530
[YUNIKORN-3134] Publish design doc (#538)
Closes: #538
Signed-off-by: Manikandan R <ma...@gmail.com>
---
...tion_successful_best_effort_preemption_case.png | Bin 0 -> 39146 bytes
...quota_preemption_successful_preemption_case.png | Bin 0 -> 45757 bytes
...ota_preemption_unsuccessful_preemption_case.png | Bin 0 -> 33860 bytes
docs/design/quota_preemptor.md | 223 +++++++++++++++++++++
sidebars.js | 1 +
5 files changed, 224 insertions(+)
diff --git a/docs/assets/quota_preemption_successful_best_effort_preemption_case.png b/docs/assets/quota_preemption_successful_best_effort_preemption_case.png
new file mode 100644
index 0000000000..0f813b863e
Binary files /dev/null and b/docs/assets/quota_preemption_successful_best_effort_preemption_case.png differ
diff --git a/docs/assets/quota_preemption_successful_preemption_case.png b/docs/assets/quota_preemption_successful_preemption_case.png
new file mode 100644
index 0000000000..8efda70774
Binary files /dev/null and b/docs/assets/quota_preemption_successful_preemption_case.png differ
diff --git a/docs/assets/quota_preemption_unsuccessful_preemption_case.png b/docs/assets/quota_preemption_unsuccessful_preemption_case.png
new file mode 100644
index 0000000000..1eacf6bb6c
Binary files /dev/null and b/docs/assets/quota_preemption_unsuccessful_preemption_case.png differ
diff --git a/docs/design/quota_preemptor.md b/docs/design/quota_preemptor.md
new file mode 100644
index 0000000000..2edc8e6fe4
--- /dev/null
+++ b/docs/design/quota_preemptor.md
@@ -0,0 +1,223 @@
+---
+id: quota_change_enforcement_through_preemption
+title: Quota Enforcement through Preemption
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Motivation
+
+Queue quota can be increased or decreased based on the need. In case of an increase, the scheduler works as normal and no change in behavior. As usual, it keeps permitting both the new and existing apps to run based on the newly configured quota. In case of an decrease, the scheduler stops accepting new workloads and waits for the running applications' natural exit so that newly configured quota comes into effect eventually. This behavior of natural exit is acceptable as long all the run [...]
+
+## Goals
+
+* Implement Preemption only for quota decrease.
+* Honour Queue Guaranteed Resources
+* Inter-Queue Preemption configuration should not be used in conjunction
+* Queue Priority related configurations should not be used in conjunction
+
+## Non-Goals
+
+* Intra-Queue Preemption
+* Cross Node Preemption
+
+## Quota Preemption Configuration
+
+Quota Enforcement through Preemption feature can be turned off globally by setting the
+appropriate property at partition level. It is configurable as follows:
+
+```yaml
+partitions:
+ - name: <name of the partition>
+ preemption:
+ quotapreemptionenabled: <boolean value>
+ queues:
+ - name: <name of the queue>
+ resources:
+ max: <maximum resources allowed for this queue>
+```
+
+Default is false (disabled). It means, the whole feature is turned OFF globally and preemption won’t be triggered when lowering the quota which is nothing but the situation exists today. So, having default as false is to retain the existing behavior as is.
+
+Setting it to true would turn ON the feature globally and preemption would be triggered whenever any queue quota decreases.
+
+## Quota Preemption Delay
+
+Quota Preemption Delay is the time duration after which the preemption should get triggered for quota changes. Otherwise, triggering preemption immediately has a profound impact especially on the queues where long live applications run. It is applicable only for quota decrease, nothing to do with increase. It is configurable.Unit is seconds.
+
+It is configurable as follows:
+
+```yaml
+partitions:
+ - name: <name of the partition>
+ queues:
+ - name: <name of the queue>
+ resources:
+ max: <maximum resources allowed for this queue>
+ quota.preemption.delay: <quota preemption delay in seconds>
+```
+It could be any value between 0 and maxint.
+
+The default is 0. It means, preemption won’t be triggered when lowering the quota which is nothing but the situation exists today. As of now, In case of quota decrease, new quota would be applied and brought into effect only for the new requests during the next scheduling cycle but existing applications continue to run as is. So, having default as 0 is to retain the existing behavior as is.
+
+Setting any value between 60 seconds and 5 hours is preferable for most of the cases.
+
+Examples are
+
+1\) To trigger preemption after 2 hours
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: queueA
+ resources:
+ max:
+ {memory: 10G}
+ quota.preemption.delay: 7200
+```
+
+2\) To trigger preemption immediately after 5 minutes
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: queueA
+ resources:
+ max:
+ {memory: 10G}
+ quota.preemption.delay: 5
+```
+
+The delay timer clocks in once the config map has been saved. Config map could be modified for multiple scenarios, but the timer clocks in only when either Queue max resources or quota preemption delay property changes. Once the timer started, changes could happen to either one of these two properties. In case of any such changes, timers reset and start again from the beginning, not on top of the current timer value to avoid unnecessary confusions. Manipulation based on the current timer [...]
+In addition, changes could be made to more than one queue at the same time but with different delay values. In case of the same delay for leaf queue and other parent queue in the whole queue hierarchy, leaf queue could be prioritized over the parent as it might help the [Queue selection process](#queue-selection-and-ordering) for the parent as described later. Need not to go through further on this in detail here as these cases would be sorted out appropriately during the implementation.
+
+### Impact of Restart
+
+How does Quota Preemption Delay work after the quota changes followed up by restart? As explained earlier, the clock kicks in once the config map has been saved, say T1. Yunikorn restarted at T2. When service starts again after T2, Quota Preemption Delay would start again from the beginning and lead to postponing this activity due to lost time (T2-T1) as opposed to earlier schedule.
+
+## Meaningful Quota Decrease
+
+When Queue has both max resources and guaranteed resources set, decreasing quota to lower value should always be greater than guaranteed resources. Otherwise, it would violate the [goal](#goals) of considering guaranteed resources also into account and making sure usage doesn't fall below guaranteed resources as part of the preemption process. Preemption should not make usage fall below guaranteed resources at any cost. Config validation is required to ensure max resource is always great [...]
+
+## Preemptable Resource calculation
+
+Whenever any queue quota changes, the first thing would be to understand the usage. If the usage is below or equal to the new quota, no need to trigger preemption. Otherwise, preemption would be triggered. We need to calculate the resources that need to be preemptable in order to bring newer quotas into effect. So, a preemptable resource is the amount of resources that needs to be preempted in order to bring the new quota into effect immediately.
+
+Preemptable resource “preemptable resource” would be derived by subtracting the current usage from newly defined “max resources” quota using subtract method
+
+resources.subOnlyExisting(existing, newer)
+
+as resource types might differ. Different possible scenarios are lowering all resource types, lowering specific resource types, complete elimination of specific resource types from its earlier configurations, brand new resource types not present in its earlier configurations etc. Using the above said resource subtraction method would guarantee the reliable delta calculation.
+
+Results could contain all positive values or negative values or combinations of both. Results derived containing negative value for any resource type are the only ones that need to be preempted. Nothing needs to be done for all other resource types as usage should be either equal or below the corresponding resource types in ““max resources” value. So, “preemptable resource” contains only resource types which need to be preempted.
+
+Below table covers the various possible scenarios for the queue `root.a.b` with examples:
+
+|Queue| Current Max | New Max | Usage | Preemptable Resources |
+|--|-----------------------|-----------------------|-----------------------|-----------------------|
+| `root.a.b` | \{M: 100G\} | \{M: 50G\} | \{M: 80G\} | \{M: 30G\} |
+| | Nil | \{M: 50G\} | \{M: 80G\} | \{M: 30G\} |
+| | \{M: 100G, CPU: 100\} | \{M: 50G, CPU: 50\} | \{M: 80G, CPU: 80\} | \{M: 30G, CPU: 30\} |
+| | \{M: 100G, CPU: 100\} | \{M:200G, CPU: 50\} | \{M: 100G, CPU: 80\} | \{CPU: 30\} |
+| | \{M: 100G\} | \{M: 100G, CPU: 100\} | \{M: 50G, CPU: 500\} | \{CPU: 400\} |
+| | \{M: 100G, CPU: 100\} | \{M: 50G\} | \{M: 80G, CPU: 100\} | \{M: 30G\} |
+| | \{M: 100G\} | \{CPU: 100\} | \{M: 100G, CPU: 500\} | \{CPU: 400\} |
+
+## Lowering Leaf Queue Quota
+
+Whenever any leaf queue quota changes, the above described preemptable resource should be used as the target. Once we know the target, the next step is to go over the tasks running in the queue being worked upon to choose the potential victims.
+
+### How are victims selected and processed?
+
+Collect all running tasks based on below described criteria:
+
+1. Ignore Daemon set pods
+2. Choose pods which have “at least one” match with preemptable resources.
+
+Once the set of tasks been collected, same has to be sorted based on the below described preference rules:
+
+1. Prefer Non originator pods over originator
+2. In case of tie in rule \#1, prefer pods based on priority. Lower priority picked up first followed by high priority pods.
+3. In case of tie in rule \#2, Prefer pods for which the “allowPreemption” flag has been set to true over other pods.
+4. In case of tie in rule \#3, prefer “young” pods based on the age
+
+
+Once the tasks are sorted, tasks can be traversed one by one and preempted to free up resources until we reach the target.
+
+## Lowering Parent Queue Quota
+
+Whenever any leaf queue quota changes, we know the specific queue to work on. Whereas, in case of lowering parent queue quota, we need to traverse all the way down to collect the end leaf queues because it is the place where actual tasks need to be preempted. But how do we select the queue(s) among different queue paths located under the parent queue for which quota has been decreased?
+
+### Queue selection and ordering
+
+Child Queues with nil usage and queues for which [preemption is already in progress](#quota-preemption-delay) would be eliminated in the beginning itself to avoid participation in below described steps. Once elimination is over, an above calculated preemptable resource could be distributed fairly among the remaining child queues as described below.
+
+Usage above the guaranteed resources only is allowed to be preempted to ensure that this preemption doesn’t end up in taking usage below the guaranteed resources. So, Usage above Guaranteed resources is calculated and curved out from each child queue and termed as “releasable resources”. It is nothing but subtracting the usage from the configured guaranteed resources. Guaranteed resources might have been configured only for a few types but not for all. So, all different combinations are [...]
+
+Once “releasable resources” is derived, calculate “total releasable resources” by adding all child queues “releasable resources”. After that “releasable resources %” for each child queue could be derived by dividing its “releasable resources” by “total releasable resources”. “releasable resources %” might have different values for each resource type as it all depends on the guaranteed resources configuration and usages. Now, applying “releasable resources %” on above calculated preemptab [...]
+
+Honoring Guaranteed resources on one specific queue would increase or decrease the preemption rate on other siblings is a side effect caused by the above shift which is something admin should take into account while doing quota readjustments. We can introduce a configuration to decide the percentage of distribution among queues or queue based individual preemption property to permit the maximum preemptable resource later.
+
+Since we know the preemptable resource target for each leaf queue located under the current parent queue being worked upon, the next step is to repeat the same process for all levels until we reach the leaf queues branched from different queues at different levels. As always, a recursive approach would be followed here too. After reaching the end leaf queues, next step is to follow the steps described in [victims selection process](#how-are-victims-selected-and-processed?)
+
+Queue selection process prior to doing [victims selection process](#how-are-victims-selected-and-processed?) has a disadvantage of not doing sorting based on pod significance across all queues. However, given the advantage as described above, Queue selection ensures fairness in the preemption process triggered through lowering of parent queue quota.
+
+## “Preemption does not help” situations
+
+Resources used by each victim may or may not differ. Selecting victims from potential victims involves necessary checks, one among them is to ensure usage doesn’t go below the guaranteed resources. So, there could be chances that there are no viable options to proceed further. For example, Preemptable resources is \{memory: 40 GB\}, Guaranteed resources: \{memory: 50 GB\} and Usage is \{memory: 100 GB\}. 5 victims. Each victim is using \{memory: 20 GB\}. Preempting 2 victims is not a pr [...]
+Unlike Intra queue preemption, where in asks (preemptor) goes through preemption process every scheduling cycle with an interval of \`X\` seconds and probability of preemption yielding positive outcome after multiple attempts is higher, this preemption process doesn’t have a such inherent retry capabilities to increase the probability of successful preemption. Even if it is available, it is not going to help much as running victims are long lived workloads and not going to change immedi [...]
+
+However, there could be situations where the above explained “best effort” might not be applied to get best out of the process as it depends on the no. of potential victims available at any moment. If there are not enough victims available, maybe only 1 victim, preemption does not help at any cost as attempting further would make usage go below the guaranteed resources.
+
+Administrators should be informed about all these kinds of situations to avoid surprises through well defined mechanisms.
+
+
+
+
+
+
+
+## Relevance of existing Intra Queue Preemption Configurations and Queue Setups
+
+In short, all existing Intra Queue Preemption Configurations have no relevance from this design point of view and should not be used in conjunction with newly added configurations.
+
+### Preemption Fence & Preemption Delay
+
+Existing Preemption related properties like preemption.policy, preemption.delay should not be considered or used in conjunction with this [preemption](#motivation) as the objective is completely different from the other one. In addition, the reason for not considering preemption.policy (fence) into account especially for the child queues underneath the current parent queue being worked upon is to avoid unbiased and incorrect decisions. A fence is a unidirectional way of traversing down t [...]
+
+Queue=\>properties=\>preemption.delay should not be confused with above discussed Quota Change Preemption delay [Queue=\>resources=\>preemption.delay](#quota-preemption-configuration) and treated differently as earlier has been introduced to compensate for the scheduling cycle interval and used only in the scheduling cycle core path.
+
+### Priority Fence & Offset
+
+priority.policy (fence) & priority.offset makes an impact while working across different queues. These two properties have been used in Inter queue preemption before choosing the victims from queues located under different queue paths. Whereas, this preemption looks inside the queue or queues located underneath the parent queue being worked upon to apply quota change. So, above said priority related properties need not to be considered.
+
+### Dynamic Queues
+
+When a Dynamic queue is created, it uses the child template defined on its immediate queue or any ancestor queue in the whole queue hierarchy. When any lowering quota happens on existing child templates, the currently used dynamic queue can be left as is as the queue itself has a shorter life span, and would be deleted anyway once the application completes. For the newly created dynamic queue, resource configuration is going to be based on the newly defined quota and works as expected.
+
+Quota could be lowered through annotations as well. In such cases, default behavior would be applied as there is no way to set preemption.delay through annotations as of now. It means, a new lowered quota would be applied immediately only for new requests and existing applications should continue to run as is and go through natural exit.
+
+## Future Plans
+
+1. Victims could be further segregated based on the nodes. Once node wise victims are available, node fulfillment processes like bin packing etc could be brought into picture.
+2. Introduce a “closest match” algorithm to choose the victims which is less intrusive from a general perspective. Prefer “closest match” victim by starting with choosing victims whose resource types exactly match with “preemptable resources”, then ordered (ascending) by count of non matching resource types.
+
diff --git a/sidebars.js b/sidebars.js
index 3abb47de95..9b0c80013d 100644
--- a/sidebars.js
+++ b/sidebars.js
@@ -103,6 +103,7 @@ module.exports = {
'design/cache_removal',
'design/preemption',
'design/simple_preemptor',
+ 'design/quota_change_enforcement_through_preemption',
'design/generic_resource',
'design/priority_scheduling',
'design/resilience',
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org