You are viewing a plain text version of this content. The canonical link for it is here.
Posted to issues@yunikorn.apache.org by pb...@apache.org on 2024/05/21 09:08:41 UTC
(yunikorn-site) branch master updated: [YUNIKORN-2614] Update website for 1.5.1 (#431)
This is an automated email from the ASF dual-hosted git repository.
pbacsko pushed a commit to branch master
in repository https://gitbox.apache.org/repos/asf/yunikorn-site.git
The following commit(s) were added to refs/heads/master by this push:
new 0ad23dabda [YUNIKORN-2614] Update website for 1.5.1 (#431)
0ad23dabda is described below
commit 0ad23dabdacea386bf869a7661d98713e4c00e0d
Author: Peter Bacsko <pb...@cloudera.com>
AuthorDate: Tue May 21 11:08:27 2024 +0200
[YUNIKORN-2614] Update website for 1.5.1 (#431)
Closes: #431
Signed-off-by: Peter Bacsko <pb...@cloudera.com>
---
doap_YuniKorn.rdf | 5 +
docusaurus.config.js | 2 +-
src/pages/community/download.md | 2 +-
versioned_docs/version-1.5.1/api/cluster.md | 79 +
versioned_docs/version-1.5.1/api/scheduler.md | 2140 ++++++++++++++++++++
versioned_docs/version-1.5.1/api/system.md | 225 ++
.../archived_design/cross_queue_preemption.md | 126 ++
.../version-1.5.1/archived_design/k8shim.md | 74 +
.../archived_design/namespace_resource_quota.md | 183 ++
.../archived_design/pluggable_app_management.md | 75 +
.../version-1.5.1/archived_design/predicates.md | 80 +
.../archived_design/scheduler_core_design.md | 401 ++++
.../archived_design/state_aware_scheduling.md | 112 +
.../version-1.5.1/assets/RunningSparkOnK8s.png | Bin 0 -> 25878 bytes
.../version-1.5.1/assets/allocation_4k.png | Bin 0 -> 68868 bytes
.../version-1.5.1/assets/application-state.png | Bin 0 -> 105253 bytes
.../version-1.5.1/assets/architecture.png | Bin 0 -> 188534 bytes
.../version-1.5.1/assets/cpu_profile.jpg | Bin 0 -> 259919 bytes
.../version-1.5.1/assets/dashboard_secret.png | Bin 0 -> 31815 bytes
.../assets/dashboard_token_select.png | Bin 0 -> 47533 bytes
.../assets/docker-dektop-minikube.png | Bin 0 -> 249095 bytes
.../version-1.5.1/assets/docker-desktop.png | Bin 0 -> 247891 bytes
.../version-1.5.1/assets/external_userinfo.png | Bin 0 -> 112563 bytes
.../version-1.5.1/assets/fifo-state-example.png | Bin 0 -> 239302 bytes
.../version-1.5.1/assets/gang_clean_up.png | Bin 0 -> 169887 bytes
.../version-1.5.1/assets/gang_generic_flow.png | Bin 0 -> 222524 bytes
.../version-1.5.1/assets/gang_scheduling_intro.png | Bin 0 -> 43907 bytes
.../version-1.5.1/assets/gang_timeout.png | Bin 0 -> 179466 bytes
.../version-1.5.1/assets/gang_total_ask.png | Bin 0 -> 22557 bytes
.../version-1.5.1/assets/goland_debug.jpg | Bin 0 -> 198845 bytes
.../version-1.5.1/assets/goland_debug.png | Bin 0 -> 290778 bytes
.../version-1.5.1/assets/grafana_dashboard.png | Bin 0 -> 215232 bytes
.../version-1.5.1/assets/grafana_login_page.png | Bin 0 -> 1785895 bytes
.../version-1.5.1/assets/import_dashboard_01.png | Bin 0 -> 701141 bytes
.../version-1.5.1/assets/import_dashboard_02.png | Bin 0 -> 283108 bytes
.../assets/k8shim-application-state.png | Bin 0 -> 129912 bytes
.../version-1.5.1/assets/k8shim-task-state.png | Bin 0 -> 144820 bytes
.../version-1.5.1/assets/kwok-affinity.png | Bin 0 -> 10716 bytes
.../version-1.5.1/assets/kwok-priorityclass.png | Bin 0 -> 19425 bytes
.../version-1.5.1/assets/kwok-throughput-01.png | Bin 0 -> 19471 bytes
.../version-1.5.1/assets/kwok-throughput-02.png | Bin 0 -> 19317 bytes
.../version-1.5.1/assets/kwok-throughput-03.png | Bin 0 -> 18247 bytes
.../version-1.5.1/assets/kwok-toleration-taint.png | Bin 0 -> 10541 bytes
.../version-1.5.1/assets/namespace-mapping.png | Bin 0 -> 327547 bytes
.../version-1.5.1/assets/node-bin-packing.png | Bin 0 -> 232909 bytes
versioned_docs/version-1.5.1/assets/node-fair.png | Bin 0 -> 462310 bytes
.../version-1.5.1/assets/node_fairness_conf.png | Bin 0 -> 12287 bytes
.../version-1.5.1/assets/object-state.png | Bin 0 -> 39732 bytes
.../version-1.5.1/assets/perf-tutorial-build.png | Bin 0 -> 53518 bytes
.../assets/perf-tutorial-resultDiagrams.png | Bin 0 -> 251263 bytes
.../assets/perf-tutorial-resultLog.png | Bin 0 -> 204787 bytes
.../version-1.5.1/assets/perf_e2e_test.png | Bin 0 -> 3957 bytes
.../version-1.5.1/assets/perf_e2e_test_conf.png | Bin 0 -> 25200 bytes
.../version-1.5.1/assets/perf_node_fairness.png | Bin 0 -> 26614 bytes
.../version-1.5.1/assets/perf_throughput.png | Bin 0 -> 13024 bytes
.../version-1.5.1/assets/pluggable-app-mgmt.jpg | Bin 0 -> 79170 bytes
.../version-1.5.1/assets/predicateComaparation.png | Bin 0 -> 182417 bytes
.../version-1.5.1/assets/predicate_4k.png | Bin 0 -> 86281 bytes
.../version-1.5.1/assets/preemption_fence.png | Bin 0 -> 105855 bytes
.../assets/preemption_fence_case1.png | Bin 0 -> 12023 bytes
.../assets/preemption_fence_case2.png | Bin 0 -> 19244 bytes
.../assets/preemption_fence_hierarchy_case.png | Bin 0 -> 63151 bytes
.../assets/preemption_normal_case.png | Bin 0 -> 18914 bytes
.../assets/preemption_priority_queue_case.png | Bin 0 -> 23392 bytes
.../assets/preemption_priorityclass_case.png | Bin 0 -> 25626 bytes
.../version-1.5.1/assets/priority-flows.png | Bin 0 -> 233513 bytes
.../version-1.5.1/assets/priority-tree.png | Bin 0 -> 142135 bytes
.../version-1.5.1/assets/prometheus-cmd.png | Bin 0 -> 542991 bytes
.../version-1.5.1/assets/prometheus-web-ui.png | Bin 0 -> 324110 bytes
versioned_docs/version-1.5.1/assets/prometheus.png | Bin 0 -> 88021 bytes
.../version-1.5.1/assets/queue-fairness.png | Bin 0 -> 173299 bytes
.../version-1.5.1/assets/queue-resource-quotas.png | Bin 0 -> 283689 bytes
.../version-1.5.1/assets/quota_check.png | Bin 0 -> 235186 bytes
.../assets/resilience-node-recovery.jpg | Bin 0 -> 319477 bytes
.../version-1.5.1/assets/resilience-workflow.jpg | Bin 0 -> 441551 bytes
.../assets/scheduling_no_predicate_4k.png | Bin 0 -> 96404 bytes
.../assets/scheduling_with_predicate_4k_.png | Bin 0 -> 107557 bytes
.../version-1.5.1/assets/simple_preemptor.png | Bin 0 -> 81655 bytes
.../version-1.5.1/assets/sparkResult.png | Bin 0 -> 38170 bytes
.../version-1.5.1/assets/tf-job-gpu-on-logs.png | Bin 0 -> 221691 bytes
.../version-1.5.1/assets/tf-job-gpu-on-ui.png | Bin 0 -> 101211 bytes
.../version-1.5.1/assets/tf-job-on-ui.png | Bin 0 -> 327800 bytes
versioned_docs/version-1.5.1/assets/throughput.png | Bin 0 -> 252615 bytes
.../version-1.5.1/assets/throughput_3types.png | Bin 0 -> 173025 bytes
.../version-1.5.1/assets/throughput_conf.png | Bin 0 -> 26837 bytes
.../version-1.5.1/assets/translationBuilding.png | Bin 0 -> 152552 bytes
.../version-1.5.1/assets/translationDemo.png | Bin 0 -> 4243960 bytes
versioned_docs/version-1.5.1/assets/ug_lookup.png | Bin 0 -> 106036 bytes
.../version-1.5.1/assets/yk-ui-screenshots.gif | Bin 0 -> 1126148 bytes
.../assets/yunikorn-gpu-time-slicing.png | Bin 0 -> 40653 bytes
.../version-1.5.1/assets/yunirkonVSdefault.png | Bin 0 -> 156100 bytes
.../version-1.5.1/design/architecture.md | 62 +
.../version-1.5.1/design/cache_removal.md | 456 +++++
versioned_docs/version-1.5.1/design/config_v2.md | 346 ++++
.../version-1.5.1/design/gang_scheduling.md | 605 ++++++
.../version-1.5.1/design/generic_resource.md | 69 +
.../design/historical_usage_tracking.md | 759 +++++++
.../design/interface_message_simplification.md | 309 +++
versioned_docs/version-1.5.1/design/preemption.md | 581 ++++++
.../version-1.5.1/design/priority_scheduling.md | 408 ++++
versioned_docs/version-1.5.1/design/resilience.md | 144 ++
.../design/scheduler_configuration.md | 246 +++
.../version-1.5.1/design/scheduler_plugin.md | 112 +
.../version-1.5.1/design/simple_preemptor.md | 114 ++
.../version-1.5.1/design/user_group_enforcement.md | 478 +++++
.../version-1.5.1/design/user_group_handling.md | 290 +++
.../version-1.5.1/design/user_group_manager.md | 410 ++++
.../version-1.5.1/developer_guide/build.md | 244 +++
.../version-1.5.1/developer_guide/dependencies.md | 124 ++
.../version-1.5.1/developer_guide/deployment.md | 145 ++
.../version-1.5.1/developer_guide/e2e_test.md | 97 +
.../version-1.5.1/developer_guide/env_setup.md | 299 +++
.../developer_guide/openshift_development.md | 189 ++
.../developer_guide/scheduler_object_states.md | 113 ++
.../version-1.5.1/get_started/core_features.md | 101 +
.../version-1.5.1/get_started/get_started.md | 83 +
.../version-1.5.1/get_started/version.md | 58 +
versioned_docs/version-1.5.1/metrics/queue.md | 110 +
versioned_docs/version-1.5.1/metrics/runtime.md | 39 +
versioned_docs/version-1.5.1/metrics/scheduler.mdx | 314 +++
.../evaluate_perf_function_with_kubemark.md | 120 ++
.../evaluate_perf_function_with_kwok.md | 173 ++
.../version-1.5.1/performance/metrics.md | 105 +
.../performance/performance_tutorial.md | 522 +++++
.../version-1.5.1/performance/profiling.md | 122 ++
versioned_docs/version-1.5.1/user_guide/acls.md | 119 ++
.../version-1.5.1/user_guide/deployment_modes.md | 69 +
.../version-1.5.1/user_guide/event_system.md | 147 ++
.../version-1.5.1/user_guide/gang_scheduling.md | 297 +++
.../labels_and_annotations_in_yunikorn.md | 52 +
.../version-1.5.1/user_guide/placement_rules.md | 355 ++++
.../version-1.5.1/user_guide/preemption.md | 231 +++
.../version-1.5.1/user_guide/priorities.md | 220 ++
.../version-1.5.1/user_guide/prometheus.md | 169 ++
.../version-1.5.1/user_guide/queue_config.md | 464 +++++
.../user_guide/resource_quota_mgmt.md | 348 ++++
.../version-1.5.1/user_guide/service_config.md | 1131 +++++++++++
.../version-1.5.1/user_guide/sorting_policies.md | 162 ++
.../version-1.5.1/user_guide/troubleshooting.md | 227 +++
.../version-1.5.1/user_guide/use_case.md | 483 +++++
.../user_guide/usergroup_resolution.md | 120 ++
.../user_guide/workloads/run_flink.md | 66 +
.../version-1.5.1/user_guide/workloads/run_mpi.md | 112 +
.../user_guide/workloads/run_nvidia.md | 353 ++++
.../user_guide/workloads/run_spark.md | 166 ++
.../user_guide/workloads/run_tensorflow.md | 228 +++
.../user_guide/workloads/workload_overview.md | 60 +
versioned_sidebars/version-1.5.1-sidebars.json | 108 +
versions.json | 1 +
149 files changed, 17537 insertions(+), 2 deletions(-)
diff --git a/doap_YuniKorn.rdf b/doap_YuniKorn.rdf
index 27945faf2a..0d46056084 100644
--- a/doap_YuniKorn.rdf
+++ b/doap_YuniKorn.rdf
@@ -35,6 +35,11 @@
<programming-language>JavaScript</programming-language>
<category rdf:resource="https://projects.apache.org/category/cloud" />
<release>
+ <Version>
+ <name>YuniKorn 1.5.1</name>
+ <created>2024-05-16</created>
+ <revision>1.5.1</revision>
+ </Version>
<Version>
<name>YuniKorn 1.5.0</name>
<created>2024-03-14</created>
diff --git a/docusaurus.config.js b/docusaurus.config.js
index 7b7f694fa4..3d41bfbd88 100644
--- a/docusaurus.config.js
+++ b/docusaurus.config.js
@@ -49,7 +49,7 @@ module.exports = {
announcementBar: {
id: 'new_release',
content:
- '1.5.0 has been released, check the <a href="/community/download">DOWNLOADS</a>.',
+ '1.5.1 has been released, check the <a href="/community/download">DOWNLOADS</a>.',
backgroundColor: '#fafbfc',
textColor: '#091E42',
},
diff --git a/src/pages/community/download.md b/src/pages/community/download.md
index 683a866263..17e99907ca 100644
--- a/src/pages/community/download.md
+++ b/src/pages/community/download.md
@@ -35,7 +35,7 @@ The latest release of Apache YuniKorn is v1.5.0.
| Version | Release date | Source download | Docker images [...]
|---------|--------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-------------------------------------------------------------------------------------------------------------------------------------------------------------------------- [...]
-| v1.5.0 | 2024-03-14 | [Download](https://www.apache.org/dyn/closer.lua/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz)<br/>[Checksum](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.sha512) & [Signature](https://downloads.apache.org/yunikorn/1.5.0/apache-yunikorn-1.5.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.5.0/images/sha256-9cefd0df164b9c4d39f9e10b010eaf7d8f89b130de1648e94f75b9b95d300a00)<br/>[admission- [...]
+| v1.5.1 | 2024-05-16 | [Download](https://www.apache.org/dyn/closer.lua/yunikorn/1.5.1/apache-yunikorn-1.5.1-src.tar.gz)<br/>[Checksum](https://downloads.apache.org/yunikorn/1.5.1/apache-yunikorn-1.5.1-src.tar.gz.sha512) & [Signature](https://downloads.apache.org/yunikorn/1.5.1/apache-yunikorn-1.5.1-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.5.1/images/sha256-dc8ef00adc8be927e83f4b9c168edd4fc881e3f1dcf39016f534e610acd31d52)<br/>[admission- [...]
| v1.4.0 | 2023-11-20 | [Download](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz)<br/>[Checksum](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.4.0/apache-yunikorn-1.4.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.4.0/images/sha256-d013be8e3ad7eb8e51ce23951e6899a4b74088e52c3767f3fcc7efcdcc0904f5)<br/>[admission- [...]
| v1.3.0 | 2023-06-12 | [Download](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz)<br/>[Checksum](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.sha512) & [Signature](https://archive.apache.org/dist/yunikorn/1.3.0/apache-yunikorn-1.3.0-src.tar.gz.asc) | [scheduler](https://hub.docker.com/layers/apache/yunikorn/scheduler-1.3.0/images/sha256-99a1973728c6684b1da7631dbf015daa1dbf519dbab1ffc8b23fccdfa7ffd0c5)<br/>[admission- [...]
diff --git a/versioned_docs/version-1.5.1/api/cluster.md b/versioned_docs/version-1.5.1/api/cluster.md
new file mode 100644
index 0000000000..35e98dac44
--- /dev/null
+++ b/versioned_docs/version-1.5.1/api/cluster.md
@@ -0,0 +1,79 @@
+---
+id: cluster
+title: Cluster
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Clusters
+
+Returns general information about the clusters managed by the YuniKorn Scheduler.
+The response includes build information about resource managers.
+
+**URL** : `/ws/v1/clusters`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+As an example, here is a response from a cluster with 1 resource manager.
+
+```json
+[
+ {
+ "startTime": 1697100824863892713,
+ "rmBuildInformation": [
+ {
+ "arch": "amd64",
+ "buildDate": "2023-09-04T18:11:43+0800",
+ "buildVersion": "latest",
+ "coreSHA": "0ecf24d2aad2",
+ "goVersion": "1.21",
+ "isPluginVersion": "false",
+ "rmId": "mycluster",
+ "shimSHA": "8b26c373b4b5",
+ "siSHA": "e7622cf54e95"
+ }
+ ],
+ "partition": "default",
+ "clusterName": "kubernetes"
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
diff --git a/versioned_docs/version-1.5.1/api/scheduler.md b/versioned_docs/version-1.5.1/api/scheduler.md
new file mode 100644
index 0000000000..2c0f2c98c4
--- /dev/null
+++ b/versioned_docs/version-1.5.1/api/scheduler.md
@@ -0,0 +1,2140 @@
+---
+id: scheduler
+title: Scheduler
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Overview
+
+The scheduler REST API returns information about various objects used by the YuniKorn Scheduler.
+
+Many of these APIs return collections of resources. Internally, all resources are represented as raw
+64-bit signed integer types. When interpreting responses from the REST API, resources of type `memory`
+are returned in units of bytes while resources of type `vcore` are returned in units of millicores
+(thousands of a core). All other resource types have no specific unit assigned.
+
+Under the `allocations` field in the response content for the app/node-related calls in the following spec, `placeholderUsed` refers to whether or not the allocation is a replacement for a placeholder. If true, `requestTime` is the creation time of its placeholder allocation, otherwise it's that of the allocation's ask. `allocationTime` is the creation time of the allocation, and `allocationDelay` is simply the difference between `allocationTime` and `requestTime`.
+
+## Partitions
+
+Returns general information and statistics about a partition.
+
+**URL** : `/ws/v1/partitions`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+[
+ {
+ "clusterId": "mycluster",
+ "name": "default",
+ "state": "Active",
+ "lastStateTransitionTime": 1649167576110754000,
+ "capacity": {
+ "capacity": {
+ "ephemeral-storage": 188176871424,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 1000000000,
+ "pods": 330,
+ "vcore": 1000
+ },
+ "usedCapacity": {
+ "memory": 800000000,
+ "vcore": 500
+ },
+ "utilization": {
+ "memory": 80,
+ "vcore": 50
+ }
+ },
+ "nodeSortingPolicy": {
+ "type": "fair",
+ "resourceWeights": {
+ "memory": 1.5,
+ "vcore": 1.3
+ }
+ },
+ "applications": {
+ "New": 5,
+ "Pending": 5,
+ "total": 10
+ },
+ "totalContainers": 0,
+ "totalNodes": 2
+ },
+ {
+ "clusterId": "mycluster",
+ "name": "gpu",
+ "state": "Active",
+ "lastStateTransitionTime": 1649167576111236000,
+ "capacity": {
+ "capacity": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "usedCapacity": {
+ "memory": 500000000,
+ "vcore": 300
+ },
+ "utilization": {
+ "memory": 25,
+ "vcore": 15
+ }
+ },
+ "nodeSortingPolicy": {
+ "type": "binpacking",
+ "resourceWeights": {
+ "memory": 0,
+ "vcore": 4.11
+ }
+ },
+ "applications": {
+ "New": 5,
+ "Running": 10,
+ "Pending": 5,
+ "total": 20
+ },
+ "totalContainers": 20,
+ "totalNodes": 5
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Queues
+
+### Partition queues
+
+Fetch all Queues associated with given Partition and displays general information about the queues like name, status, capacities and properties.
+The queues' hierarchy is kept in the response json.
+
+**URL** : `/ws/v1/partition/{partitionName}/queues`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+For the default queue hierarchy (only `root.default` leaf queue exists) a similar response to the following is sent back to the client:
+
+```json
+[
+ {
+ "queuename": "root",
+ "status": "Active",
+ "maxResource": {
+ "ephemeral-storage": 188176871424,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 8000000000,
+ "pods": 330,
+ "vcore": 8000
+ },
+ "guaranteedResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "allocatedResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "pendingResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "isLeaf": "false",
+ "isManaged": "false",
+ "properties": {
+ "application.sort.policy": "stateaware"
+ },
+ "parent": "",
+ "template": {
+ "maxResource": {
+ "memory": 8000000000,
+ "vcore": 8000
+ },
+ "guaranteedResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "properties": {
+ "application.sort.policy": "stateaware"
+ }
+ },
+ "partition": "default",
+ "children": [
+ {
+ "queuename": "root.default",
+ "status": "Active",
+ "maxResource": {
+ "memory": 8000000000,
+ "vcore": 8000
+ },
+ "guaranteedResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "allocatedResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "pendingResource": {
+ "memory": 54000000,
+ "vcore": 80
+ },
+ "isLeaf": "true",
+ "isManaged": "false",
+ "properties": {
+ "application.sort.policy": "stateaware"
+ },
+ "parent": "root",
+ "template": null,
+ "children": [],
+ "absUsedCapacity": {
+ "memory": 1,
+ "vcore": 0
+ },
+ "maxRunningApps": 12,
+ "runningApps": 4,
+ "allocatingAcceptedApps": [
+ "app-1",
+ "app-2"
+ ]
+ }
+ ],
+ "absUsedCapacity": {
+ "memory": 1,
+ "vcore": 0
+ }
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Applications
+
+### Partition applications
+
+Fetch all Applications for the given Partition/State combination and displays general information about the applications like used resources, queue name, submission time and allocations.
+Only following application states are allowed: active, rejected, completed. Active is a fake state that represents all application states except completed and rejected.
+For active state, can narrow the result by status query parameters(case-insensitive). For example, can fetch `Running` applications for the default partition by
+`/ws/v1/partition/default/applications/active?status=running`.
+
+**URL** : `/ws/v1/partition/:partition/applications/:state`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+The content of the application object is the same as Queue Applications. See
+ [Queue Applications](#queue-applications) for details.
+
+### Queue applications
+
+Fetch all Applications for the given Partition/Queue combination and displays general information about the applications like used resources, queue name, submission time and allocations.
+
+**URL** : `/ws/v1/partition/{partitionName}/queue/{queueName}/applications`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Deprecated**:
+
+Field `uuid` has been deprecated, would be removed from below response in YUNIKORN 1.7.0 release. `AllocationID` has replaced `uuid`. Both `uuid` and `AllocationID` fields have the same value. `AllocationID` has extra suffix containing hyphen and counter (-0, -1 and so on) at the end.
+
+**Content examples**
+
+In the example below there are three allocations belonging to two applications, one with a pending request.
+
+```json
+[
+ {
+ "applicationID": "application-0001",
+ "usedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "maxUsedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "pendingResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "partition": "default",
+ "queueName": "root.default",
+ "submissionTime": 1648754032076020293,
+ "requests": [
+ {
+ "allocationKey": "f137fab6-3cfa-4536-93f7-bfff92689382",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task2"
+ },
+ "requestTime": 16487540320812345678,
+ "resource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "pendingCount": 1,
+ "priority": "0",
+ "requiredNodeId": "",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderTimeout": 0,
+ "taskGroupName": "",
+ "allocationLog": [
+ {
+ "message": "node(s) didn't match Pod's node affinity, node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate",
+ "lastOccurrence": 16487540320812346001,
+ "count": 81
+ },
+ {
+ "message": "node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, node(s) didn't match Pod's node affinity",
+ "lastOccurrence": 16487540320812346002,
+ "count": 504
+ },
+ {
+ "message": "node(s) didn't match Pod's node affinity",
+ "lastOccurrence": 16487540320812346003,
+ "count": 1170
+ }
+ ]
+ }
+ ],
+ "allocations": [
+ {
+ "allocationKey": "deb12221-6b56-4fe9-87db-ebfadce9aa20",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task0"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "9af35d44-2d6f-40d1-b51d-758859e6b8a8",
+ "allocationID": "9af35d44-2d6f-40d1-b51d-758859e6b8a8-0",
+ "resource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "priority": "0",
+ "nodeId": "node-0001",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": true
+ }
+ ],
+ "applicationState": "Running",
+ "user": "system:serviceaccount:kube-system:deployment-controller",
+ "groups": [
+ "system:serviceaccounts",
+ "system:serviceaccounts:kube-system",
+ "system:authenticated"
+ ],
+ "rejectedMessage": "",
+ "stateLog": [
+ {
+ "time": 1648741409145224000,
+ "applicationState": "Accepted"
+ },
+ {
+ "time": 1648741409147432100,
+ "applicationState": "Running"
+ }
+ ],
+ "placeholderData": [
+ {
+ "taskGroupName": "task-group-example",
+ "count": 2,
+ "minResource": {
+ "memory": 1000000000,
+ "vcore": 100
+ },
+ "replaced": 1,
+ "timedout": 1
+ }
+ ],
+ "hasReserved": false,
+ "reservations": []
+ },
+ {
+ "applicationID": "application-0002",
+ "usedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "maxUsedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "pendingResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "partition": "default",
+ "queueName": "root.default",
+ "submissionTime": 1648754032076020293,
+ "requests": [],
+ "allocations": [
+ {
+ "allocationKey": "54e5d77b-f4c3-4607-8038-03c9499dd99d",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0002",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task0"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "08033f9a-4699-403c-9204-6333856b41bd",
+ "allocationID": "08033f9a-4699-403c-9204-6333856b41bd-0",
+ "resource": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "priority": "0",
+ "nodeId": "node-0001",
+ "applicationId": "application-0002",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": false
+ },
+ {
+ "allocationKey": "af3bd2f3-31c5-42dd-8f3f-c2298ebdec81",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0002",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task1"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "96beeb45-5ed2-4c19-9a83-2ac807637b3b",
+ "allocationID": "96beeb45-5ed2-4c19-9a83-2ac807637b3b-0",
+ "resource": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "priority": "0",
+ "nodeId": "node-0002",
+ "applicationId": "application-0002",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": false
+ }
+ ],
+ "applicationState": "Running",
+ "user": "system:serviceaccount:kube-system:deployment-controller",
+ "groups": [
+ "system:serviceaccounts",
+ "system:serviceaccounts:kube-system",
+ "system:authenticated"
+ ],
+ "rejectedMessage": "",
+ "stateLog": [
+ {
+ "time": 1648741409145224000,
+ "applicationState": "Accepted"
+ },
+ {
+ "time": 1648741409147432100,
+ "applicationState": "Running"
+ }
+ ],
+ "placeholderData": [],
+ "hasReserved": false,
+ "reservations": []
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Application
+
+### Partition/Queue application
+
+Fetch an Application given a Partition, Queue(optional) and Application ID and displays general information about the application like used resources, queue name, submission time and allocations.
+
+**URL** : `/ws/v1/partition/{partitionName}/application/{appId}` or `/ws/v1/partition/{partitionName}/queue/{queueName}/application/{appId}`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Deprecated**:
+
+Field `uuid` has been deprecated, would be removed from below response in YUNIKORN 1.7.0 release. `AllocationID` has replaced `uuid`. Both `uuid` and `AllocationID` fields have the same value. `AllocationID` has extra suffix containing hyphen and counter (-0, -1 and so on) at the end.
+
+**Content example**
+
+```json
+{
+ "applicationID": "application-0001",
+ "usedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "maxUsedResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "pendingResource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "partition": "default",
+ "queueName": "root.default",
+ "submissionTime": 1648754032076020293,
+ "requests": [
+ {
+ "allocationKey": "f137fab6-3cfa-4536-93f7-bfff92689382",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task2"
+ },
+ "requestTime": 16487540320812345678,
+ "resource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "pendingCount": 1,
+ "priority": "0",
+ "requiredNodeId": "",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderTimeout": 0,
+ "taskGroupName": "",
+ "allocationLog": [
+ {
+ "message": "node(s) didn't match Pod's node affinity, node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate",
+ "lastOccurrence": 16487540320812346001,
+ "count": 81
+ },
+ {
+ "message": "node(s) had taint {node-role.kubernetes.io/master: }, that the pod didn't tolerate, node(s) didn't match Pod's node affinity",
+ "lastOccurrence": 16487540320812346002,
+ "count": 504
+ },
+ {
+ "message": "node(s) didn't match Pod's node affinity",
+ "lastOccurrence": 16487540320812346003,
+ "count": 1170
+ }
+ ]
+ }
+ ],
+ "allocations": [
+ {
+ "allocationKey": "deb12221-6b56-4fe9-87db-ebfadce9aa20",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task0"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "9af35d44-2d6f-40d1-b51d-758859e6b8a8",
+ "allocationID": "9af35d44-2d6f-40d1-b51d-758859e6b8a8-0",
+ "resource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "priority": "0",
+ "nodeId": "node-0001",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": true
+ }
+ ],
+ "applicationState": "Running",
+ "user": "system:serviceaccount:kube-system:deployment-controller",
+ "groups": [
+ "system:serviceaccounts",
+ "system:serviceaccounts:kube-system",
+ "system:authenticated"
+ ],
+ "rejectedMessage": "",
+ "stateLog": [
+ {
+ "time": 1648741409145224000,
+ "applicationState": "Accepted"
+ },
+ {
+ "time": 1648741409147432100,
+ "applicationState": "Running"
+ }
+ ],
+ "placeholderData": [
+ {
+ "taskGroupName": "task-group-example",
+ "count": 2,
+ "minResource": {
+ "memory": 1000000000,
+ "vcore": 100
+ },
+ "replaced": 1,
+ "timedout": 1
+ }
+ ],
+ "hasReserved": false,
+ "reservations": []
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## UsersTracker
+### Get users usage tracking information
+
+Fetch all users usage given a Partition and displays general information about the users managed by YuniKorn.
+
+**URL** : `/ws/v1/partition/{partitionName}/usage/users`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content example**
+
+```json
+[
+ {
+ "userName": "user1",
+ "groups": {
+ "app2": "tester"
+ },
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": [
+ "app2"
+ ],
+ "children": []
+ }]
+ }
+ },
+ {
+ "userName": "user2",
+ "groups": {
+ "app1": "tester"
+ },
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 11000000000,
+ "vcore": 10000
+ },
+ "runningApplications": ["app1", "app2", "app3"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 5000000000,
+ "vcore": 5000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "runningApplications": [
+ "app3"
+ ],
+ "children": []
+ }]
+ }
+ }
+]
+```
+
+### Error response
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## UserTracker
+### Get specific user usage tracking information
+Fetch specific user usage given a Partition and displays general information about the users managed by YuniKorn.
+
+**URL** : `/ws/v1/partition/{partitionName}/usage/user/{userName}`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content example**
+
+```json
+{
+ "userName": "user1",
+ "groups": {
+ "app1": "tester"
+ },
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": [
+ "app2"
+ ],
+ "children": []
+ }]
+ }
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## GroupsTracker
+### Get groups usage tracking information
+Fetch all groups usage given a Partition and displays general information about the groups managed by YuniKorn.
+
+**URL** : `/ws/v1/partition/{partitionName}/usage/groups`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content example**
+
+```json
+[
+ {
+ "groupName": "group1",
+ "applications": ["app1", "app2"],
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": [
+ "app2"
+ ],
+ "children": []
+ }]
+ }
+ },
+ {
+ "groupName": "group2",
+ "applications": ["app1", "app2", "app3"],
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 11000000000,
+ "vcore": 10000
+ },
+ "runningApplications": ["app1", "app2", "app3"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 5000000000,
+ "vcore": 5000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "runningApplications": [
+ "app3"
+ ],
+ "children": []
+ }]
+ }
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## GroupTracker
+### Get specific group usage tracking information
+
+Fetch specific group usage given a Partition and displays general information about the groups managed by YuniKorn.
+
+**URL** : `/ws/v1/partition/{partitionName}/usage/group/{groupName}`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content example**
+
+```json
+{
+ "groupName": "group1",
+ "applications": ["app1", "app2"],
+ "queues":
+ {
+ "queuePath": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuePath": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuePath": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": [
+ "app2"
+ ],
+ "children": []
+ }]
+ }
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Nodes
+
+### Partition nodes
+
+Fetch all Nodes associated with given Partition and displays general information about the nodes managed by YuniKorn.
+Node details include host and rack name, capacity, resources, utilization, and allocations.
+
+**URL** : `/ws/v1/partition/{partitionName}/nodes`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+Here you can see an example response from a 2-node cluster having 3 allocations.
+
+```json
+[
+ {
+ "nodeID": "node-0001",
+ "hostName": "",
+ "rackName": "",
+ "attributes": {
+ "beta.kubernetes.io/arch": "amd64",
+ "beta.kubernetes.io/os": "linux",
+ "kubernetes.io/arch": "amd64",
+ "kubernetes.io/hostname": "node-0001",
+ "kubernetes.io/os": "linux",
+ "node-role.kubernetes.io/control-plane": "",
+ "node-role.kubernetes.io/master": "",
+ "node.kubernetes.io/exclude-from-external-load-balancers": "",
+ "ready": "true",
+ "si.io/hostname": "node-0001",
+ "si.io/rackname": "/rack-default",
+ "si/instance-type": "",
+ "si/node-partition": "[mycluster]default"
+ },
+ "capacity": {
+ "ephemeral-storage": 75850798569,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 14577000000,
+ "pods": 110,
+ "vcore": 10000
+ },
+ "allocated": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "occupied": {
+ "memory": 154000000,
+ "vcore" :750
+ },
+ "available": {
+ "ephemeral-storage": 75850798569,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 6423000000,
+ "pods": 110,
+ "vcore": 1250
+ },
+ "utilized": {
+ "memory": 3,
+ "vcore": 13
+ },
+ "allocations": [
+ {
+ "allocationKey": "54e5d77b-f4c3-4607-8038-03c9499dd99d",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task0"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "08033f9a-4699-403c-9204-6333856b41bd",
+ "allocationID": "08033f9a-4699-403c-9204-6333856b41bd-0",
+ "resource": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "priority": "0",
+ "nodeId": "node-0001",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": false
+ },
+ {
+ "allocationKey": "deb12221-6b56-4fe9-87db-ebfadce9aa20",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0002",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task0"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "9af35d44-2d6f-40d1-b51d-758859e6b8a8",
+ "allocationID": "9af35d44-2d6f-40d1-b51d-758859e6b8a8-0",
+ "resource": {
+ "memory": 4000000000,
+ "vcore": 4000
+ },
+ "priority": "0",
+ "nodeId": "node-0001",
+ "applicationId": "application-0002",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": false
+ }
+ ],
+ "schedulable": true
+ },
+ {
+ "nodeID": "node-0002",
+ "hostName": "",
+ "rackName": "",
+ "attributes": {
+ "beta.kubernetes.io/arch": "amd64",
+ "beta.kubernetes.io/os": "linux",
+ "kubernetes.io/arch": "amd64",
+ "kubernetes.io/hostname": "node-0002",
+ "kubernetes.io/os": "linux",
+ "ready": "false",
+ "si.io/hostname": "node-0002",
+ "si.io/rackname": "/rack-default",
+ "si/instance-type": "",
+ "si/node-partition": "[mycluster]default"
+ },
+ "capacity": {
+ "ephemeral-storage": 75850798569,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 14577000000,
+ "pods": 110,
+ "vcore": 10000
+ },
+ "allocated": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "occupied": {
+ "memory": 154000000,
+ "vcore" :750
+ },
+ "available": {
+ "ephemeral-storage": 75850798569,
+ "hugepages-1Gi": 0,
+ "hugepages-2Mi": 0,
+ "memory": 6423000000,
+ "pods": 110,
+ "vcore": 1250
+ },
+ "utilized": {
+ "memory": 8,
+ "vcore": 38
+ },
+ "allocations": [
+ {
+ "allocationKey": "af3bd2f3-31c5-42dd-8f3f-c2298ebdec81",
+ "allocationTags": {
+ "kubernetes.io/label/app": "sleep",
+ "kubernetes.io/label/applicationId": "application-0001",
+ "kubernetes.io/label/queue": "root.default",
+ "kubernetes.io/meta/namespace": "default",
+ "kubernetes.io/meta/podName": "task1"
+ },
+ "requestTime": 1648754034098912461,
+ "allocationTime": 1648754035973982920,
+ "allocationDelay": 1875070459,
+ "uuid": "96beeb45-5ed2-4c19-9a83-2ac807637b3b",
+ "allocationID": "96beeb45-5ed2-4c19-9a83-2ac807637b3b-0",
+ "resource": {
+ "memory": 2000000000,
+ "vcore": 2000
+ },
+ "priority": "0",
+ "nodeId": "node-0002",
+ "applicationId": "application-0001",
+ "partition": "default",
+ "placeholder": false,
+ "placeholderUsed": false
+ }
+ ],
+ "schedulable": true,
+ "isReserved": false,
+ "reservations": []
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Node
+
+### Partition node
+
+Fetch a Node associated with given Partition and Node ID and displays general information about the node managed by YuniKorn.
+Node details include host and rack name, capacity, resources, utilization, and allocations.
+
+**URL** : `/ws/v1/partition/{partitionName}/node/{nodeId}`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+{
+ "nodeID":"node-0001",
+ "hostName":"",
+ "rackName":"",
+ "capacity":{
+ "ephemeral-storage":75850798569,
+ "hugepages-1Gi":0,
+ "hugepages-2Mi":0,
+ "memory":14577000000,
+ "pods":110,
+ "vcore":10000
+ },
+ "allocated":{
+ "memory":6000000000,
+ "vcore":6000
+ },
+ "occupied":{
+ "memory":154000000,
+ "vcore":750
+ },
+ "available":{
+ "ephemeral-storage":75850798569,
+ "hugepages-1Gi":0,
+ "hugepages-2Mi":0,
+ "memory":6423000000,
+ "pods":110,
+ "vcore":1250
+ },
+ "utilized":{
+ "memory":3,
+ "vcore":13
+ },
+ "allocations":[
+ {
+ "allocationKey":"54e5d77b-f4c3-4607-8038-03c9499dd99d",
+ "allocationTags":{
+ "kubernetes.io/label/app":"sleep",
+ "kubernetes.io/label/applicationId":"application-0001",
+ "kubernetes.io/label/queue":"root.default",
+ "kubernetes.io/meta/namespace":"default",
+ "kubernetes.io/meta/podName":"task0"
+ },
+ "requestTime":1648754034098912461,
+ "allocationTime":1648754035973982920,
+ "allocationDelay":1875070459,
+ "uuid":"08033f9a-4699-403c-9204-6333856b41bd",
+ "allocationID":"08033f9a-4699-403c-9204-6333856b41bd-0",
+ "resource":{
+ "memory":2000000000,
+ "vcore":2000
+ },
+ "priority":"0",
+ "nodeId":"node-0001",
+ "applicationId":"application-0001",
+ "partition":"default",
+ "placeholder":false,
+ "placeholderUsed":false
+ },
+ {
+ "allocationKey":"deb12221-6b56-4fe9-87db-ebfadce9aa20",
+ "allocationTags":{
+ "kubernetes.io/label/app":"sleep",
+ "kubernetes.io/label/applicationId":"application-0002",
+ "kubernetes.io/label/queue":"root.default",
+ "kubernetes.io/meta/namespace":"default",
+ "kubernetes.io/meta/podName":"task0"
+ },
+ "requestTime":1648754034098912461,
+ "allocationTime":1648754035973982920,
+ "allocationDelay":1875070459,
+ "uuid":"9af35d44-2d6f-40d1-b51d-758859e6b8a8",
+ "allocationID":"9af35d44-2d6f-40d1-b51d-758859e6b8a8-0",
+ "resource":{
+ "memory":4000000000,
+ "vcore":4000
+ },
+ "priority":"0",
+ "nodeId":"node-0001",
+ "applicationId":"application-0002",
+ "partition":"default",
+ "placeholder":false,
+ "placeholderUsed":false
+ }
+ ],
+ "schedulable":true
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Node utilization
+
+Show how every node is distributed with regard to dominant resource utilization.
+
+**Status** : Deprecated since v1.5.0 and will be removed in the next major release. Replaced with `/ws/v1/scheduler/node-utilizations`.
+
+**URL** : `/ws/v1/scheduler/node-utilization`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+{
+ "type": "vcore",
+ "utilization": [
+ {
+ "bucketName": "0-10%",
+ "numOfNodes": 1,
+ "nodeNames": [
+ "aethergpu"
+ ]
+ },
+ {
+ "bucketName": "10-20%",
+ "numOfNodes": 2,
+ "nodeNames": [
+ "primary-node",
+ "second-node"
+ ]
+ },
+ ...
+ ]
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Node utilizations
+
+Show the nodes utilization of different types of resources in a cluster.
+
+**URL** : `/ws/v1/scheduler/node-utilizations`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+[
+ {
+ "clusterId": "mycluster",
+ "partition": "default",
+ "utilizations": [
+ {
+ "type": "pods",
+ "utilization": [
+ {
+ "bucketName": "0-10%",
+ "numOfNodes": 2,
+ "nodeNames": [
+ "primary-node",
+ "second-node"
+ ]
+ },
+ {
+ "bucketName": "10-20%"
+ },
+ ...
+ ]
+ },
+ {
+ "type": "vcores",
+ "utilization": [
+ {
+ "bucketName": "0-10%",
+ "numOfNodes": 2,
+ "nodeNames": [
+ "primary-node",
+ "second-node"
+ ]
+ },
+ {
+ "bucketName": "10-20%"
+ },
+ ...
+ ]
+ },
+ ...
+ ]
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Goroutines info
+
+Dumps the stack traces of the currently running goroutines.
+
+**URL** : `/ws/v1/stack`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```text
+goroutine 356 [running
+]:
+github.com/apache/yunikorn-core/pkg/webservice.getStackInfo.func1(0x30a0060,
+0xc003e900e0,
+0x2)
+ /yunikorn/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20200717041747-f3e1c760c714/pkg/webservice/handlers.go: 41 +0xab
+github.com/apache/yunikorn-core/pkg/webservice.getStackInfo(0x30a0060,
+0xc003e900e0,
+0xc00029ba00)
+ /yunikorn/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20200717041747-f3e1c760c714/pkg/webservice/handlers.go: 48 +0x71
+net/http.HandlerFunc.ServeHTTP(0x2df0e10,
+0x30a0060,
+0xc003e900e0,
+0xc00029ba00)
+ /usr/local/go/src/net/http/server.go: 1995 +0x52
+github.com/apache/yunikorn-core/pkg/webservice.Logger.func1(0x30a0060,
+0xc003e900e0,
+0xc00029ba00)
+ /yunikorn/go/pkg/mod/github.com/apache/yunikorn-core@v0.0.0-20200717041747-f3e1c760c714/pkg/webservice/webservice.go: 65 +0xd4
+net/http.HandlerFunc.ServeHTTP(0xc00003a570,
+0x30a0060,
+0xc003e900e0,
+0xc00029ba00)
+ /usr/local/go/src/net/http/server.go: 1995 +0x52
+github.com/gorilla/mux.(*Router).ServeHTTP(0xc00029cb40,
+0x30a0060,
+0xc003e900e0,
+0xc0063fee00)
+ /yunikorn/go/pkg/mod/github.com/gorilla/mux@v1.7.3/mux.go: 212 +0x140
+net/http.serverHandler.ServeHTTP(0xc0000df520,
+0x30a0060,
+0xc003e900e0,
+0xc0063fee00)
+ /usr/local/go/src/net/http/server.go: 2774 +0xcf
+net/http.(*conn).serve(0xc0000eab40,
+0x30a61a0,
+0xc003b74000)
+ /usr/local/go/src/net/http/server.go: 1878 +0x812
+created by net/http.(*Server).Serve
+ /usr/local/go/src/net/http/server.go: 2884 +0x4c5
+
+goroutine 1 [chan receive,
+ 26 minutes
+]:
+main.main()
+ /yunikorn/pkg/shim/main.go: 52 +0x67a
+
+goroutine 19 [syscall,
+ 26 minutes
+]:
+os/signal.signal_recv(0x1096f91)
+ /usr/local/go/src/runtime/sigqueue.go: 139 +0x9f
+os/signal.loop()
+ /usr/local/go/src/os/signal/signal_unix.go: 23 +0x30
+created by os/signal.init.0
+ /usr/local/go/src/os/signal/signal_unix.go: 29 +0x4f
+
+...
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Metrics
+
+Endpoint to retrieve metrics from the Prometheus server.
+The metrics are dumped with help messages and type information.
+
+**URL** : `/ws/v1/metrics`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```text
+# HELP go_gc_duration_seconds A summary of the pause duration of garbage collection cycles.
+# TYPE go_gc_duration_seconds summary
+go_gc_duration_seconds{quantile="0"} 2.567e-05
+go_gc_duration_seconds{quantile="0.25"} 3.5727e-05
+go_gc_duration_seconds{quantile="0.5"} 4.5144e-05
+go_gc_duration_seconds{quantile="0.75"} 6.0024e-05
+go_gc_duration_seconds{quantile="1"} 0.00022528
+go_gc_duration_seconds_sum 0.021561648
+go_gc_duration_seconds_count 436
+# HELP go_goroutines Number of goroutines that currently exist.
+# TYPE go_goroutines gauge
+go_goroutines 82
+# HELP go_info Information about the Go environment.
+# TYPE go_info gauge
+go_info{version="go1.12.17"} 1
+# HELP go_memstats_alloc_bytes Number of bytes allocated and still in use.
+# TYPE go_memstats_alloc_bytes gauge
+go_memstats_alloc_bytes 9.6866248e+07
+
+...
+
+# HELP yunikorn_scheduler_vcore_nodes_usage Nodes resource usage, by resource name.
+# TYPE yunikorn_scheduler_vcore_nodes_usage gauge
+yunikorn_scheduler_vcore_nodes_usage{range="(10%, 20%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(20%,30%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(30%,40%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(40%,50%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(50%,60%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(60%,70%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(70%,80%]"} 1
+yunikorn_scheduler_vcore_nodes_usage{range="(80%,90%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="(90%,100%]"} 0
+yunikorn_scheduler_vcore_nodes_usage{range="[0,10%]"} 0
+```
+
+## Configuration validation
+
+**URL** : `/ws/v1/validate-conf`
+
+**Method** : `POST`
+
+**Auth required** : NO
+
+### Success response
+
+Regardless whether the configuration is allowed or not if the server was able to process the request, it will yield a 200 HTTP status code.
+
+**Code** : `200 OK`
+
+#### Allowed configuration
+
+Sending the following simple configuration yields an accept
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: test
+```
+
+Reponse
+
+```json
+{
+ "allowed": true,
+ "reason": ""
+}
+```
+
+#### Disallowed configuration
+
+The following configuration is not allowed due to the "wrong_text" field put into the yaml file.
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ queues:
+ - name: test
+ - wrong_text
+```
+
+Reponse
+
+```json
+{
+ "allowed": false,
+ "reason": "yaml: unmarshal errors:\n line 7: cannot unmarshal !!str `wrong_text` into configs.PartitionConfig"
+}
+```
+
+## Configuration
+
+Endpoint to retrieve the current scheduler configuration
+
+**URL** : `/ws/v1/config`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content example (with `Accept: application/json` header)**
+
+```json
+{
+ "Partitions": [
+ {
+ "Name": "default",
+ "Queues": [
+ {
+ "Name": "root",
+ "Parent": true,
+ "Resources": {},
+ "SubmitACL": "*",
+ "ChildTemplate": {
+ "Resources": {}
+ }
+ }
+ ],
+ "PlacementRules": [
+ {
+ "Name": "tag",
+ "Create": true,
+ "Filter": {
+ "Type": ""
+ },
+ "Value": "namespace"
+ }
+ ],
+ "Preemption": {
+ "Enabled": false
+ },
+ "NodeSortPolicy": {
+ "Type": ""
+ }
+ }
+ ],
+ "Checksum": "FD5D3726DF0F02416E02F3919D78F61B15D14425A34142D93B24C137ED056946",
+ "Extra": {
+ "event.trackingEnabled": "false",
+ "log.core.scheduler.level": "info",
+ "log.core.security.level": "info",
+ "log.level": "debug"
+ }
+}
+```
+
+**Content example (without `Accept: application/json` header)**
+
+```yaml
+partitions:
+ - name: default
+ queues:
+ - name: root
+ parent: true
+ submitacl: "*"
+ placementrules:
+ - name: tag
+ create: true
+ value: namespace
+checksum: FD5D3726DF0F02416E02F3919D78F61B15D14425A34142D93B24C137ED056946
+extra:
+ event.trackingEnabled: "false"
+ log.core.scheduler.level: info
+ log.core.security.level: info
+ log.level: debug
+
+```
+
+## Application history
+
+Endpoint to retrieve historical data about the number of total applications by timestamp.
+
+**URL** : `/ws/v1/history/apps`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+[
+ {
+ "timestamp": 1595939966153460000,
+ "totalApplications": "1"
+ },
+ {
+ "timestamp": 1595940026152892000,
+ "totalApplications": "1"
+ },
+ {
+ "timestamp": 1595940086153799000,
+ "totalApplications": "2"
+ },
+ {
+ "timestamp": 1595940146154497000,
+ "totalApplications": "2"
+ },
+ {
+ "timestamp": 1595940206155187000,
+ "totalApplications": "2"
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+## Container history
+
+Endpoint to retrieve historical data about the number of total containers by timestamp.
+
+**URL** : `/ws/v1/history/containers`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+[
+ {
+ "timestamp": 1595939966153460000,
+ "totalContainers": "1"
+ },
+ {
+ "timestamp": 1595940026152892000,
+ "totalContainers": "1"
+ },
+ {
+ "timestamp": 1595940086153799000,
+ "totalContainers": "3"
+ },
+ {
+ "timestamp": 1595940146154497000,
+ "totalContainers": "3"
+ },
+ {
+ "timestamp": 1595940206155187000,
+ "totalContainers": "3"
+ }
+]
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+**Content examples**
+
+```json
+{
+ "status_code": 500,
+ "message": "system error message. for example, json: invalid UTF-8 in string: ..",
+ "description": "system error message. for example, json: invalid UTF-8 in string: .."
+}
+```
+
+
+## Endpoint healthcheck
+
+Endpoint to retrieve historical data about critical logs, negative resource on node/cluster/app, ...
+
+**URL** : `/ws/v1/scheduler/healthcheck`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```json
+{
+ "Healthy": true,
+ "HealthChecks": [
+ {
+ "Name": "Scheduling errors",
+ "Succeeded": true,
+ "Description": "Check for scheduling error entries in metrics",
+ "DiagnosisMessage": "There were 0 scheduling errors logged in the metrics"
+ },
+ {
+ "Name": "Failed nodes",
+ "Succeeded": true,
+ "Description": "Check for failed nodes entries in metrics",
+ "DiagnosisMessage": "There were 0 failed nodes logged in the metrics"
+ },
+ {
+ "Name": "Negative resources",
+ "Succeeded": true,
+ "Description": "Check for negative resources in the partitions",
+ "DiagnosisMessage": "Partitions with negative resources: []"
+ },
+ {
+ "Name": "Negative resources",
+ "Succeeded": true,
+ "Description": "Check for negative resources in the nodes",
+ "DiagnosisMessage": "Nodes with negative resources: []"
+ },
+ {
+ "Name": "Consistency of data",
+ "Succeeded": true,
+ "Description": "Check if a node's allocated resource <= total resource of the node",
+ "DiagnosisMessage": "Nodes with inconsistent data: []"
+ },
+ {
+ "Name": "Consistency of data",
+ "Succeeded": true,
+ "Description": "Check if total partition resource == sum of the node resources from the partition",
+ "DiagnosisMessage": "Partitions with inconsistent data: []"
+ },
+ {
+ "Name": "Consistency of data",
+ "Succeeded": true,
+ "Description": "Check if node total resource = allocated resource + occupied resource + available resource",
+ "DiagnosisMessage": "Nodes with inconsistent data: []"
+ },
+ {
+ "Name": "Consistency of data",
+ "Succeeded": true,
+ "Description": "Check if node capacity >= allocated resources on the node",
+ "DiagnosisMessage": "Nodes with inconsistent data: []"
+ },
+ {
+ "Name": "Reservation check",
+ "Succeeded": true,
+ "Description": "Check the reservation nr compared to the number of nodes",
+ "DiagnosisMessage": "Reservation/node nr ratio: [0.000000]"
+ }
+ ]
+}
+```
+
+## Retrieve full state dump
+
+Endpoint to retrieve the following information in a single response:
+
+* Current timestamp (Unix timestamp in nanosecond)
+* List of partitions
+* List of applications (running, completed and rejected)
+* Application history
+* Nodes
+* Generic cluster information
+* Container history
+* Queues
+* RMDiagnostics
+* Log level
+* Configuration
+
+**URL** : `/ws/v1/fullstatedump`
+
+**Method** : `GET`
+
+**Auth required** : NO
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+The output of this REST query can be rather large, and it is a combination of those which have already been demonstrated.
+
+The `RMDiagnostics` shows the content of the K8Shim cache. The exact content is version dependent and has not stabilised.
+The current content shows the cached objects:
+* nodes
+* pods
+* priorityClasses
+* schedulingState (pod status)
+
+### Failure response
+
+**Code**: `500 Internal Server Error`
+
+## Batch Events
+
+Endpoint is used to retrieve a batch of event records.
+
+**URL**: `/ws/v1/events/batch`
+
+**METHOD** : `GET`
+
+**Auth required** : NO
+
+**URL query parameters** :
+- `count` (optional) : Specifies the maxmem number of events to be included in the response.
+- `start` (optional) : Specifies the starting ID for retrieving events. If the specified ID is outside the ring buffer
+(too low or too high), the response will include the lowest and highest ID values with `EventRecords` being empty.
+
+
+### Success response
+
+**Code**: `200 OK`
+
+**Content examples**
+
+```json
+{
+ "InstanceUUID": "400046c6-2180-41a2-9be1-1c251ab2c498",
+ "LowestID": 0,
+ "HighestID": 7,
+ "EventRecords": [
+ {
+ "type": 3,
+ "objectID": "yk8s-worker",
+ "message": "schedulable: true",
+ "timestampNano": 1701347180239597300,
+ "eventChangeType": 1,
+ "eventChangeDetail": 302,
+ "resource": {}
+ },
+ {
+ "type": 3,
+ "objectID": "yk8s-worker",
+ "message": "Node added to the scheduler",
+ "timestampNano": 1701347180239650600,
+ "eventChangeType": 2,
+ "resource": {
+ "resources": {
+ "ephemeral-storage": {
+ "value": 502921060352
+ },
+ "hugepages-1Gi": {},
+ "hugepages-2Mi": {},
+ "memory": {
+ "value": 33424998400
+ },
+ "pods": {
+ "value": 110
+ },
+ "vcore": {
+ "value": 8000
+ }
+ }
+ }
+ }
+ ]
+}
+```
+
+### Error response
+
+**Code** : `500 Internal Server Error`
+
+## Event stream
+
+Creates a persistent HTTP connection for event streaming. New events are sent to the clients immediately, so unlike the batch interface, there is no need for polling.
+The number of active connections is limited. The default setting is 100 connections total and 15 connections per host. The respective configmap properties are `event.maxStreams` and `event.maxStreamsPerHost`.
+
+**URL**: `/ws/v1/events/stream`
+
+**METHOD** : `GET`
+
+**Auth required** : NO
+
+**URL query parameters**:
+- `count` (optional) : Specifies the number of past events (those which have been generated before the connection establishment) to include in the response. Default value is 0.
+
+### Success response
+
+**Code**: `200 OK`
+
+**Content examples**
+
+```json
+{"type":2,"objectID":"app-1","timestampNano":1708465452903045265,"eventChangeType":1,"eventChangeDetail":204,"resource":{}}
+{"type":2,"objectID":"app-1","timestampNano":1708465452903192898,"eventChangeType":2,"eventChangeDetail":201,"referenceID":"alloc-1","resource":{"resources":{"memory":{"value":10000000},"vcore":{"value":1000}}}}
+{"type":3,"objectID":"node-1:1234","timestampNano":1708465452903312146,"eventChangeType":2,"eventChangeDetail":303,"referenceID":"alloc-1","resource":{"resources":{"memory":{"value":10000000},"vcore":{"value":1000}}}}
+{"type":2,"objectID":"app-1","timestampNano":1708465452903474210,"eventChangeType":1,"eventChangeDetail":205,"resource":{}}
+{"type":5,"objectID":"testuser","timestampNano":1708465452903506166,"eventChangeType":2,"eventChangeDetail":603,"referenceID":"root.singleleaf","resource":{"resources":{"memory":{"value":10000000},"vcore":{"value":1000}}}}
+```
+
+### Error responses
+
+**Code** : `400 Bad Request` (URL query is invalid)
+**Code** : `503 Service Unavailable` (Too many active streaming connections)
+**Code** : `500 Internal Server Error`
diff --git a/versioned_docs/version-1.5.1/api/system.md b/versioned_docs/version-1.5.1/api/system.md
new file mode 100644
index 0000000000..1d685ffa58
--- /dev/null
+++ b/versioned_docs/version-1.5.1/api/system.md
@@ -0,0 +1,225 @@
+---
+id: system
+title: System
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+These endpoints are for the [pprof](https://github.com/google/pprof) profiling tool.
+
+## pprof
+
+**URL** : `/debug/pprof/`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```text
+/debug/pprof/
+
+Types of profiles available:
+Count Profile
+273 allocs
+0 block
+0 cmdline
+78 goroutine
+273 heap
+0 mutex
+0 profile
+29 threadcreate
+0 trace
+full goroutine stack dump
+Profile Descriptions:
+
+allocs: A sampling of all past memory allocations
+block: Stack traces that led to blocking on synchronization primitives
+cmdline: The command line invocation of the current program
+goroutine: Stack traces of all current goroutines
+heap: A sampling of memory allocations of live objects. You can specify the gc GET parameter to run GC before taking the heap sample.
+mutex: Stack traces of holders of contended mutexes
+profile: CPU profile. You can specify the duration in the seconds GET parameter. After you get the profile file, use the go tool pprof command to investigate the profile.
+threadcreate: Stack traces that led to the creation of new OS threads
+trace: A trace of execution of the current program. You can specify the duration in the seconds GET parameter. After you get the trace file, use the go tool trace command to investigate the trace.
+```
+
+## Heap
+
+**URL** : `/debug/pprof/heap`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Thread create
+
+**URL** : `/debug/pprof/threadcreate`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Goroutine
+
+**URL** : `/debug/pprof/goroutine`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Allocations
+
+**URL** : `/debug/pprof/allocs`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Block
+
+**URL** : `/debug/pprof/block`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Mutex
+
+**URL** : `/debug/pprof/mutex`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Cmdline
+
+**URL** : `/debug/pprof/cmdline`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Profile
+
+**URL** : `/debug/pprof/profile`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Symbol
+
+**URL** : `/debug/pprof/symbol`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
+
+## Trace
+
+**URL** : `/debug/pprof/trace`
+
+**Method** : `GET`
+
+### Success response
+
+**Code** : `200 OK`
+
+**Content examples**
+
+```proto
+// binary data from proto
+```
diff --git a/versioned_docs/version-1.5.1/archived_design/cross_queue_preemption.md b/versioned_docs/version-1.5.1/archived_design/cross_queue_preemption.md
new file mode 100644
index 0000000000..51c803308a
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/cross_queue_preemption.md
@@ -0,0 +1,126 @@
+---
+id: cross_queue_preemption
+title: Cross Queue Preemption
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Problems:
+
+According to lessons we learned from YARN Scheduler preemption.
+
+**Here're top bad things:**
+
+- Preemption is a shotgun instead of a sniper, when a preemption decision is made, nobody knows if preempted resources will go to demanding queue/app/user or not.
+- Preemption logic and allocation is different, we have to implement (and mimic) what we have done in scheduler allocation logic.
+
+**Here're top good things:**
+
+- Preemption is fast (thanks to the shotgun), reclaiming thousands of containers only takes ~ 1 sec.
+- We have understand how painful it is to handle DRF, multiple preemption policies (inter/intra-queue, shotgun/surgical preemption, etc.) And we have developed some good logic
+to make sure better modularization and plug-ability
+
+## Answer some questions for design/implementation choices
+
+**1\. Do we really want preemption-delay? (Or we just want to control pace)**
+
+In CS, we have preemption-delay, which select victims in preemption candidates, wait for a certain time before killing it.
+
+The purposes of preemption delay are: a. give heads-up time to apps so
+they can prepare bad things happen (unfortunately no app do anything for these heads up, at least from what I knew). b. control preemption pace.
+
+And in practice, I found it causes a lot of issues, for example when a
+cluster state keep changing, it is very hard to ensure accurate preemption.
+
+**Proposal:**
+
+Remove the preemption-delay, keep the logics of controlling preemption pace. (such as ```yarn.resourcemanager.monitor.capacity.preemption
+.total_preemption_per_round```). And we can do allocation together with preemption.
+This don't mean containers will be stopped immediately after preemption issued. Instead, RM can control delays between signal a container and kill a container. Such as grace
+termination of POD in K8s: https://kubernetes.io/docs/concepts/workloads/pods/pod/#termination-of-pods
+
+**2\. Do we want to do preemption for every scheduling logic, or we can do periodically?**
+
+In CS, we have preemption logic runs periodically, like every 1 sec or 3 sec.
+
+Since preemption logic involves some heavy logics, like calculating shares of queues/apps. And when doing accurate preemption, we may need to scan nodes for preemption candidate.
+Considering this, I propose to have preemption runs periodically. But it is important to note that, we need to try to use as much code as possible for
+allocation-inside-preemption, otherwise there will be too much duplicated logic and very hard to be maintained in the future.
+
+**3\. Preemption cost and function**
+
+We found it is helpful to add cost for preemption, such as container live time, priority, type of container. It could be a cost function (Which returns a numeric value) or it
+could be a comparator (which compare two allocations for preemption ask).
+
+## Pseudo code
+
+Logic of allocation (invoked every allocation cycle)
+
+```
+input:
+ - nAlloc, allocate N allocations for this allocation cycle.
+
+for partition:
+ askCandidates := findAskCandidates(nAlloc, preemption=false)
+
+ allocated, failed_to_allocated := tryAllocate(askCandidates);
+
+ send-allocated-to-cache-to-commit;
+
+ update-missed-opportunity (allocated, failed_to_allocated);
+
+ nAlloc -= len(allocated)
+```
+
+Logic of preemption (invoked every preemption cycle)
+
+```
+// It has to be done for every preemption-policy because calculation is different.
+for preemption-policy:
+ preempt_results := policy.preempt()
+ for preempt_results:
+ send-preempt-result-to-cache-to-commit;
+ updated-missed-opportunity (allocated)
+```
+
+Inside preemption policy
+
+```
+inter-queue-preempt-policy:
+ calculate-preemption-quotas;
+
+ for partitions:
+ total_preempted := resource(0);
+
+ while total_preempted < partition-limited:
+ // queues will be sorted by allocating - preempting
+ // And ignore any key in preemption_mask
+ askCandidates := findAskCandidates(N, preemption=true)
+
+ preempt_results := tryAllocate(askCandidates, preemption=true);
+
+ total_preempted += sigma(preempt_result.allocResource)
+
+ send-allocated-to-cache-to-commit;
+
+ update-missed-opportunity (allocated, failed_to_allocated);
+
+ update-preemption-mask(askCandidates.allocKeys - preempt_results.allocKeys)
+```
\ No newline at end of file
diff --git a/versioned_docs/version-1.5.1/archived_design/k8shim.md b/versioned_docs/version-1.5.1/archived_design/k8shim.md
new file mode 100644
index 0000000000..01f08db2d9
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/k8shim.md
@@ -0,0 +1,74 @@
+---
+id: k8shim
+title: Kubernetes Shim Design
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Github repo: https://github.com/apache/yunikorn-k8shim
+
+Please read the [architecture](../design/architecture.md) doc before reading this one, you will need to understand
+the 3 layer design of YuniKorn before getting to understand what is the Kubernetes shim.
+
+## The Kubernetes shim
+
+The YuniKorn Kubernetes shim is responsible for talking to Kubernetes, it is responsible for translating the Kubernetes
+cluster resources, and resource requests via scheduler interface and send them to the scheduler core.
+And when a scheduler decision is made, it is responsible for binding the pod to the specific node. All the communication
+between the shim and the scheduler core is through the scheduler-interface.
+
+## The admission controller
+
+The admission controller runs in a separate pod, it runs a
+[mutation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#mutatingadmissionwebhook)
+and a [validation webhook](https://kubernetes.io/docs/reference/access-authn-authz/admission-controllers/#validatingadmissionwebhook), where:
+
+1. The `mutation webhook` mutates pod spec by:
+ - Adding `schedulerName: yunikorn`
+ - By explicitly specifying the scheduler name, the pod will be scheduled by YuniKorn scheduler.
+ - Adding `applicationId` label
+ - When a label `applicationId` exists, reuse the given applicationId.
+ - When a label `spark-app-selector` exists, reuse the given spark app ID.
+ - Otherwise, assign a generated application ID for this pod, using convention: `yunikorn-<namespace>-autogen`. This is unique per namespace.
+ - Adding `queue` label
+ - When a label `queue` exists, reuse the given queue name. Note, if placement rule is enabled, values set in the label is ignored.
+ - Otherwise, adds `queue: root.default`
+ - Adding `disableStateAware` label
+ - If pod was assigned a generated applicationId by the admission controller, also set `disableStateAware: true`. This causes the generated application
+ to immediately transition from the `Starting` to `Running` state so that it will not block other applications.
+2. The `validation webhook` validates the configuration set in the configmap
+ - This is used to prevent writing malformed configuration into the configmap.
+ - The validation webhook calls scheduler [validation REST API](api/scheduler.md#configuration-validation) to validate configmap updates.
+
+### Admission controller deployment
+
+By default, the admission controller is deployed as part of the YuniKorn Helm chart installation. This can be disabled if necessary (though not recommended) by setting the Helm parameter `embedAdmissionController` to `false`.
+
+On startup, the admission controller performs a series of tasks to ensure that it is properly registered with Kubernetes:
+1. Loads a Kubernetes secret called `admission-controller-secrets`. This secret stores a pair of CA certificates which are used to sign the TLS server certificate used by the admission controller.
+2. If the secret cannot be found or either CA certificate is within 90 days of expiration, generates new certificate(s). If a certificate is expiring, a new one is generated with an expiration of 12 months in the future. If both certificates are missing or expiring, the second certificate is generated with an expiration of 6 months in the future. This ensures that both certificates do not expire at the same time, and that there is an overlap of trusted certificates.
+3. If the CA certificates were created or updated, writes the secrets back to Kubernetes.
+4. Generates an ephemeral TLS server certificate signed by the CA certificate with the latest expiration date.
+5. Validates, and if necessary, creates or updates the Kubernetes webhook configurations named `yunikorn-admission-controller-validations` and `yunikorn-admission-controller-mutations`. If the CA certificates have changed, the webhooks will also be updated. These webhooks allow the Kubernetes API server to connect to the admission controller service to perform configmap validations and pod mutations.
+6. Starts up the admission controller HTTPS server.
+
+Additionally, the admission controller also starts a background task to wait for CA certificates to expire. Once either certificate is expiring within the next 30 days, new CA and server certificates are generated, the webhook configurations are updated, and the HTTPS server is quickly restarted. This ensures that certificates rotate properly without downtime.
+
+In production clusters, it is recommended to deploy the admission controller with two replicas by setting the Helm parameter `admissionController.replicaCount` to `2`. This will ensure that at least one admission controller webhook is reachable by the Kubernetes API server at all times. In this configuration, the CA certificates and webhook configurations are shared between the instances.
diff --git a/versioned_docs/version-1.5.1/archived_design/namespace_resource_quota.md b/versioned_docs/version-1.5.1/archived_design/namespace_resource_quota.md
new file mode 100644
index 0000000000..90830b6a17
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/namespace_resource_quota.md
@@ -0,0 +1,183 @@
+---
+id: namespace_resource_quota
+title: Namespace Resource Quota
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+In K8s, user can setup namespace with [resource quotas](https://kubernetes.io/docs/concepts/policy/resource-quotas/) to limit aggregated resource consumption in this namespace. The validation of namespace resource quotas is handled in api-server directly, therefore YuniKorn simply honors the quotas like the default scheduler.
+
+## Best practice
+
+It is not mandatory to setup YuniKorn queues with respect of namespaces.
+However, in practice, it makes more sense to do so.
+Namespace is often used to set a cap for resource consumptions per user-group/team,
+YuniKorn queue is also meant to divide cluster resource into multiple groups.
+Let's go through an example.
+
+### 1. Setup namespace
+
+Namespace: `advertisement`:
+```
+apiVersion: v1
+kind: ResourceQuota
+metadata:
+ name: advertisement
+spec:
+ hard:
+ requests.cpu: "200m"
+ requests.memory: 2000Mi
+ limits.cpu: "200m"
+ limits.memory: 4000Mi
+```
+Create the namespace
+```
+kubectl create namespace advertisement
+kubectl create -f ./advertisement.yaml --namespace=advertisement
+kubectl get quota --namespace=advertisement
+kubectl describe quota advertisement --namespace=advertisement
+
+// output
+Name: advertisement
+Namespace: advertisement
+Resource Used Hard
+-------- ---- ----
+limits.cpu 0 200m
+limits.memory 0 4000Mi
+requests.cpu 0 200m
+requests.memory 0 2000Mi
+```
+
+### 2. Setup YuniKorn queues
+
+Queue: `advertisement`:
+```
+name: advertisement
+resources:
+ guaranteed:
+ vcore: 100
+ memory: 1000
+ max:
+ vcore: 200
+ memory: 2000
+```
+
+ensure `QueueMaxResource <= NamespaceResourceQuotaRequests`
+
+### 3. Mapping applications to queues & namespace
+
+In a pod spec
+
+```
+apiVersion: v1
+kind: Pod
+metadata:
+ namespace: advertisement
+ labels:
+ app: sleep
+ applicationId: "application_2019_01_22_00001"
+ queue: "root.advertisement"
+ name: task0
+spec:
+ schedulerName: yunikorn
+ containers:
+ - name: sleep-5s
+ image: "alpine:latest"
+ command: ["/bin/ash", "-ec", "while :; do echo '.'; sleep 5 ; done"]
+ resources:
+ requests:
+ cpu: "50m"
+ memory: "800M"
+ limits:
+ cpu: "100m"
+ memory: "1000M"
+```
+
+Check Quota
+
+```
+kubectl describe quota advertisement --namespace=advertisement
+
+Name: advertisement
+Namespace: advertisement
+Resource Used Hard
+-------- ---- ----
+limits.cpu 100m 200m
+limits.memory 1G 4000Mi
+requests.cpu 50m 200m
+requests.memory 800M 2000Mi
+```
+
+Now submit another application,
+
+```
+apiVersion: v1
+kind: Pod
+metadata:
+ namespace: advertisement
+ labels:
+ app: sleep
+ applicationId: "application_2019_01_22_00002"
+ queue: "root.advertisement"
+ name: task1
+spec:
+ schedulerName: yunikorn
+ containers:
+ - name: sleep-5s
+ image: "alpine:latest"
+ command: ["/bin/ash", "-ec", "while :; do echo '.'; sleep 5 ; done"]
+ resources:
+ requests:
+ cpu: "200m"
+ memory: "800M"
+ limits:
+ cpu: "200m"
+ memory: "1000M"
+```
+
+pod will not be able to submitted to api-server, because the requested cpu `200m` + used cpu `100m` = `300m` which exceeds the resource quota.
+
+```
+kubectl create -f pod_ns_adv_task1.yaml
+Error from server (Forbidden): error when creating "pod_ns_adv_task1.yaml": pods "task1" is forbidden: exceeded quota: advertisement, requested: limits.cpu=200m,requests.cpu=200m, used: limits.cpu=100m,requests.cpu=50m, limited: limits.cpu=200m,requests.cpu=200m
+```
+
+## Future Work
+
+For compatibility, we should respect namespaces and resource quotas.
+Resource quota is overlapped with queue configuration in many ways,
+for example the `requests` quota is just like queue's max resource. However,
+there are still a few features resource quota can do but queue cannot, such as
+
+1. Resource `limits`. The aggregated resource from all pods in a namespace cannot exceed this limit.
+2. Storage Resource Quota, e.g storage size, PVC number, etc.
+3. Object Count Quotas, e.g count of PVCs, services, configmaps, etc.
+4. Resource Quota can map to priority class.
+
+Probably we can build something similar to cover (3) in this list.
+But it would be hard to completely support all these cases.
+
+But currently, setting applications mapping to a queue as well as a corresponding namespace is over complex.
+Some future improvements might be:
+
+1. Automatically detects namespaces in k8s-shim and map them to queues. Behind the scenes, we automatically generates queue configuration based on namespace definition. Generated queues are attached under root queue.
+2. When new namespace added/updated/removed, similarly to (1), we automatically update queues.
+3. User can add more configuration to queues, e.g add queue ACL, add child queues on the generated queues.
+4. Applications submitted to namespaces are transparently submitted to corresponding queues.
\ No newline at end of file
diff --git a/versioned_docs/version-1.5.1/archived_design/pluggable_app_management.md b/versioned_docs/version-1.5.1/archived_design/pluggable_app_management.md
new file mode 100644
index 0000000000..d297adac3f
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/pluggable_app_management.md
@@ -0,0 +1,75 @@
+---
+id: pluggable_app_management
+title: Pluggable App Management
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## The Problem
+
+Currently, we schedule and group an application is based on a label on the pod.
+This generic way works for any type of workload. It does however give us a limited information on the lifecycle
+and application. On the K8s side, operators have been introduced to provide more detail on the application
+and help scheduling. We cannot use them currently and want to add that functionality.
+
+## K8s Operator Pattern
+
+[K8s operator](https://kubernetes.io/docs/concepts/extend-kubernetes/operator/)
+is a pattern in K8s to manage applications, it's a handy way to manage application's lifecycle out-of-box on K8s.
+You define several CRDs and some controllers to monitor and mutate the state of the application based on the CRD definition.
+
+For example in [spark-k8s-operator](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator),
+it defines a CRD called `SparkApplication`, the controller watches the events of add/update/delete of this CRD
+and trigger corresponding actions on event notifications. The `SparkApplication` looks like
+[this example](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/examples/spark-pi.yaml). There
+are a lot more popular operators, such as [flink-k8s-operator](https://github.com/GoogleCloudPlatform/flink-on-k8s-operator),
+ [tf-operator](https://github.com/kubeflow/tf-operator), [pytorch-operator](https://github.com/kubeflow/pytorch-operator), etc.
+
+Use Spark as an example. YuniKorn is able to schedule resources for all pods in K8s, that seamlessly supports Spark. It
+works with [native Spark on K8s](https://spark.apache.org/docs/latest/running-on-kubernetes.html), or
+[spark on K8s with operator](https://github.com/GoogleCloudPlatform/spark-on-k8s-operator/blob/master/docs/design.md#architecture),
+you'll find the difference from the design architecture chart from the given link. To support native Spark on K8s,
+YuniKorn reads pods' spec and group Spark pods by a label-selector, based on `spark-app-selector`.
+The operator approach gives us more context about the Spark job, such as a better understanding about job state.
+But all these info requires us to look at `SparkApplication` CRD, currently, there is no neat way to
+add such functionality. That's why we need to design a flexible approach to support 3rd party operators
+(retrieving info from their CRDs), so we can easily integrate with other operators with small effort.
+
+## Design
+
+The key issue here is we need a app-management interface, that can be easily extended.
+It needs to be decoupled with existing scheduling logic. For each operator, we create a service to manage this type app's lifecycle,
+and communicate with the scheduling cache independently. The high-level design looks like below:
+
+
+
+Where
+- `AppManagementService` is a composite set of services that can be managed together.
+- `AppManager` is a specific app management service for a particular type of application. In each service, it has
+ access to K8s clients, such as informers, listers, in order to monitor CRD events. And it collects necessary info
+ and talk with scheduler cache through `AMProtocol`.
+- `APIProvider` encapsulate a set of useful APIs that can be shared, such as kube-client, pod/node/storage informers, etc.
+ Each of such informers, it can be shared with multiple app managers, to avoid the overhead.
+- `AMProtocol` defines the basic interaction contract between app manager and the scheduler cache, that helps the cache
+ to performs app lifecycle management without understanding what type of the application it is.
+
+In the upon chart, the AppManagementService has 2 services, the _general_ one is managing normal applications, that
+recognizes applications by pod labels; the _spark-k8s-operator_ one watches `SparkApplication` CRD and manage jobs'
+lifecycle defined by this CRD.
\ No newline at end of file
diff --git a/versioned_docs/version-1.5.1/archived_design/predicates.md b/versioned_docs/version-1.5.1/archived_design/predicates.md
new file mode 100644
index 0000000000..9233a25f36
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/predicates.md
@@ -0,0 +1,80 @@
+---
+id: predicates
+title: Support K8s Predicates
+---
+
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+## Design
+
+Predicates are a set of pre-registered functions in K8s, the scheduler invokes these functions to check if a pod
+is eligible to be allocated onto a node. Common predicates are: node-selector, pod affinity/anti-affinity etc. To support
+these predicates in YuniKorn, we don't intend to re-implement everything on our own, but to re-use the core predicates
+code as much as possible.
+
+YuniKorn-core is agnostic about underneath RMs, so the predicates functions are implemented in K8s-shim as a `SchedulerPlugin`.
+SchedulerPlugin is a way to plug/extend scheduler capabilities. Shim can implement such plugin and register itself to
+yunikorn-core, so plugged function can be invoked in the scheduler core. Find all supported plugins in
+[types](https://github.com/apache/yunikorn-core/blob/master/pkg/plugins/types.go).
+
+## Workflow
+
+First, RM needs to register itself to yunikorn-core, it advertises what scheduler plugin interfaces are supported.
+E.g a RM could implement `PredicatePlugin` interface and register itself to yunikorn-core. Then yunikorn-core will
+call PredicatePlugin API to run predicates before making allocation decisions.
+
+
+Following workflow demonstrates how allocation looks like when predicates are involved.
+
+```
+pending pods: A, B
+shim sends requests to core, including A, B
+core starts to schedule A, B
+ partition -> queue -> app -> request
+ schedule A (1)
+ run predicates (3)
+ generate predicates metadata (4)
+ run predicate functions one by one with the metadata
+ success
+ proposal: A->N
+ schedule B (2)
+ run predicates (calling shim API)
+ generate predicates metadata
+ run predicate functions one by one with the metadata
+ success
+ proposal: B->N
+commit the allocation proposal for A and notify k8s-shim
+commit the allocation proposal for B and notify k8s-shim
+shim binds pod A to N
+shim binds pod B to N
+```
+
+(1) and (2) are running in parallel.
+
+(3) yunikorn-core calls a `schedulerPlugin` API to run predicates, this API is implemented on k8s-shim side.
+
+(4) K8s-shim generates metadata based on current scheduler cache, the metadata includes some intermittent states about nodes and pods.
+
+## Predicates White-list
+
+Intentionally, we only support a white-list of predicates. Majorly due to 2 reasons,
+* Predicate functions are time-consuming, it has negative impact on scheduler performance. To support predicates that are only necessary can minimize the impact. This will be configurable via CLI options;
+* The implementation depends heavily on K8s default scheduler code, though we reused some unit tests, the coverage is still a problem. We'll continue to improve the coverage when adding new predicates.
+
+The white-list currently is defined in [PredicateManager](https://github.com/apache/yunikorn-k8shim/blob/master/pkg/plugin/predicates/predicate_manager.go).
diff --git a/versioned_docs/version-1.5.1/archived_design/scheduler_core_design.md b/versioned_docs/version-1.5.1/archived_design/scheduler_core_design.md
new file mode 100644
index 0000000000..c3da548c03
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/scheduler_core_design.md
@@ -0,0 +1,401 @@
+---
+id: scheduler_core_design
+title: Scheduler Core Design
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+:::caution
+The scheduler core design has changed. [YUNIKORN-317](https://issues.apache.org/jira/browse/YUNIKORN-317) was committed and has removed the scheduler cache.
+This document will not be maintained and is just for historical reference.
+See [scheduler cache removal design](../design/cache_removal.md)
+:::
+
+Github repo: https://github.com/apache/yunikorn-core
+
+Scheduler core encapsulates all scheduling algorithms, it collects resources from underneath resource management
+platforms (like YARN/K8s), and is responsible for container allocation requests. It makes the decision where is the
+best spot for each request and then sends response allocations to the resource management platform.
+Scheduler core is agnostic about underneath platforms, all the communications are through the [scheduler interface](https://github.com/apache/yunikorn-scheduler-interface).
+
+## Components:
+
+```
+
+ +---------------+ +--------------+
+ |K8s Shim | |YARN Shim |
+ +---------------+ +--------------+
+
+ +--------------+ +------------+
+ Scheduler- | GRPC Protocol| |Go API |
+ Interface: +--------------+ +------------+
+
++---------------------------------------------------------------------------+
+ +--------------------+
+ |Scheduler API Server|
+ +-------------+ +---------+----------+
+ |AdminService | |
+ +-------------+ |Write Ops +----------------+
+ +-------------+ V ++Scheduler |
+ |Configurator | +-------------------+ Allocate || And |
+ +-------------+ |Cache Event Handler+<-----------------| |
+ +----------> +-------------------+ Preempt ++Preemptor |
+ Update Cfg Handled by policies +----------------+
+ + (Stateless)
+ +------v--------+
+ |Scheduler Cache|
+ +---------------+
+ +---------------------------------------------+
+ |--------+ +------+ +----------+ +----------+ |
+ ||Node | |Queue | |Allocation| |Requests | |
+ |--------+ +------+ +----------+ +----------+ |
+ +---------------------------------------------+
+```
+
+### Scheduler API Server (RMProxy)
+
+Responsible for communication between RM and Scheduler, which implements scheduler-interface GRPC protocol,
+or just APIs. (For intra-process communication w/o Serde).
+
+### Scheduler Cache
+
+Caches all data related to scheduler state, such as used resources of each queues, nodes, allocations.
+Relationship between allocations and nodes, etc. Should not include in-flight data for resource allocation.
+For example to-be-preempted allocation candidates. Fair share resource of queues, etc.
+
+### Scheduler Cache Event Handler
+
+Handles all events which needs to update scheduler internal state. So all the write operations will be carefully handled.
+
+### Admin Service
+
+Handles request from Admin, which can also load configurations from storage and update scheduler policies.
+
+### Scheduler and Preemptor
+
+Handles Scheduler's internal state. (Which is not belong to scheduelr cache), such as internal reservations, etc.
+Scheduler and preemptor will work together, make scheduling or preemption decisions. All allocate/preempt request
+will be handled by event handler.
+
+## Scheduler's responsibility
+
+- According to resource usages between queues, sort queues, applications, and figure out order of application allocation. (This will be used by preemption as well).
+- It is possible that we cannot satisfy some of the allocation request, we need to skip them and find next request.
+- It is possible that some allocation request cannot be satisfied because of resource fragmentation. We need to reserve room for such requests.
+- Different nodes may belong to different disjoint partitions, we can make independent scheduler runs
+- Be able to config and change ordering policies for apps, queues.
+- Application can choose their own way to manage sort of nodes.
+
+## Preemption
+
+- It is important to know "who wants the resource", so we can do preemption based on allocation orders.
+- When do preemption, it is also efficient to trigger allocation op. Think about how to do it.
+- Preemption needs to take care about queue resource balancing.
+
+## Communication between Shim and Core
+
+YuniKorn-Shim (like https://github.com/apache/yunikorn-k8shim) communicates with core by
+using scheduler-interface (https://github.com/apache/yunikorn-scheduler-interface).
+Scheduler interface has Go API or GRPC. Currently, yunikorn-k8shim is using Go API to communicate with yunikorn-core
+to avoid extra overhead introduced by GRPC.
+
+**Shim (like K8shim) first need to register with core:**
+
+```go
+func (m *RMProxy) RegisterResourceManager(request *si.RegisterResourceManagerRequest, callback api.ResourceManagerCallback) (*si.RegisterResourceManagerResponse, error)
+```
+
+Which indicate ResourceManager's name, a callback function for updateResponse. The design of core is be able to do scheduling for multiple clusters (such as multiple K8s cluster) just with one core instance.
+
+**Shim interacts with core by invoking RMProxy's Update API frequently, which updates new allocation request, allocation to kill, node updates, etc.**
+
+```go
+func (m *RMProxy) Update(request *si.UpdateRequest) error
+```
+
+Response of update (such as new allocated container) will be received by registered callback.
+
+## Configurations & Semantics
+
+Example of configuration:
+
+- Partition is name space.
+- Same queues can under different partitions, but enforced to have same hierarchy.
+
+ Good:
+
+ ```
+ partition=x partition=y
+ a a
+ / \ / \
+ b c b c
+ ```
+
+ Good (c in partition y acl=""):
+
+ ```
+ partition=x partition=y
+ a a
+ / \ /
+ b c b
+ ```
+
+ Bad (c in different hierarchy)
+
+ ```
+ partition=x partition=y
+ a a
+ / \ / \
+ b c b d
+ /
+ c
+ ```
+
+ Bad (Duplicated c)
+
+ ```
+ partition=x
+ a
+ / \
+ b c
+ /
+ c
+
+ ```
+
+- Different hierarchies can be added
+
+ ```scheduler-conf.yaml
+ partitions:
+ - name: default
+ queues:
+ root:
+ configs:
+ acls:
+ childrens:
+ - a
+ - b
+ - c
+ - ...
+ a:
+ configs:
+ acls:
+ capacity: (capacity is not allowed to set for root)
+ max-capacity: ...
+ mapping-policies:
+ ...
+ - name: partition_a:
+ queues:
+ root:...
+ ```
+
+## How scheduler do allocation
+
+Scheduler runs a separate goroutine to look at asks and available resources, and do resource allocation. Here's allocation logic in pseudo code:
+
+Entry point of scheduler allocation is `scheduler.go: func (s *Scheduler) schedule()`
+
+```
+# First of all, YuniKorn has partition concept, a logical resource pool can consists
+# of one of multiple physical dis-joint partitions. It is similar to YARN's node
+# partition concept.
+
+for partition : partitions:
+ # YuniKorn can reserve allocations for picky asks (such as large request, etc.)
+ # Before doing regular allocation, YuniKorn look at reservedAllocations first.
+ for reservedAllocation : partition.reservedAllocations:
+ reservedAllocation.tryAllocate(..)
+
+ # After tried all reserved allocation, YuniKorn will go to regular allocation
+ partition.tryAllocate(..)
+
+ # If there's any allocation created, scheduler will create an AllocationProposal
+ # and send to Cache to "commit" the AllocationProposal
+```
+
+**Allocation by hierchical of queues**
+
+Inside `partition.tryAllocate`
+
+It recursively traverse from root queue and down to lower level, for each level, logic is inside `pkg/scheduler/scheduling_queue.go func (sq *SchedulingQueue) tryAllocate`
+
+Remember YuniKorn natively supports hierarchical of queues. For ParentQueue (which has sub queues under the parent queue), it uses queue's own sorting policy to sort subqueues and try to allocate from most preferred queue to least-preferred queue.
+
+For LeafQueue (which has applications inside the queue), it uses queue's own sorting policy to sort applications belong to the queue and allocate based on the sorted order.
+
+(All sorting policies can be configured differently at each level.)
+
+**Allocation by application**
+
+When it goes to Application, see (`scheduler_application.go: func (sa *SchedulingApplication) tryAllocate`), It first sort the pending resource requests belong to the application (based on requests' priority). And based on the selected request, and configured node-sorting policy, it sorts nodes belong to the partition and try to allocate resources on the sorted nodes.
+
+When application trying to allocate resources on nodes, it will invokes PredicatePlugin to make sure Shim can confirm the node is good. (For example K8shim runs predicates check for allocation pre-check).
+
+**Allocation completed by scheduler**
+
+Once allocation is done, scheduler will create an AllocationProposal and send to Cache to do further check, we will cover details in the upcoming section.
+
+## Flow of events
+
+Like mentioned before, all communications between components like RMProxy/Cache/Schedulers are done by using async event handler.
+
+RMProxy/Cache/Scheduler include local event queues and event handlers. RMProxy and Scheduler have only one queue (For example: `pkg/scheduler/scheduler.go: handleSchedulerEvent`), and Cache has two queues (One for events from RMProxy, and one for events from Scheduler, which is designed for better performance).
+
+We will talk about how events flowed between components:
+
+**Events for ResourceManager registration and updates:**
+
+```
+Update from ResourceManager -> RMProxy -> RMUpdateRequestEvent Send to Cache
+New ResourceManager registration -> RMProxy -> RegisterRMEvent Send to Cache
+```
+
+**Cache Handles RM Updates**
+
+There're many fields inside RM Update event (`RMUpdateRequestEvent`), among them, we have following categories:
+
+```
+1) Update for Application-related updates
+2) Update for New allocation ask and release.
+3) Node (Such as kubelet) update (New node, remove node, node resource change, etc.)
+```
+
+More details can be found at:
+
+```
+func (m *ClusterInfo) processRMUpdateEvent(event *cacheevent.RMUpdateRequestEvent)
+
+inside cluster_info.go
+```
+
+**Cache send RM updates to Scheduler**
+
+For most cases, Cache propagate updates from RM to scheduler directly (including Application, Node, Asks, etc.). And it is possible that some updates from RM is not valid (such as adding an application to a non-existed queue), for such cases, cache can send an event back to RMProxy and notify the ResourceManager. (See `RMApplicationUpdateEvent.RejectedApplications`)
+
+**Cache handles scheduler config**
+
+Cache also handles scheduler's config changes, see
+
+```go
+func (m *ClusterInfo) processRMConfigUpdateEvent(event *commonevents.ConfigUpdateRMEvent)
+```
+
+Similar to other RM updates, it propages news to scheduelr.
+
+**Scheduler do allocation**
+
+Once an AllocationProposal created by scheduler, scheduler sends `AllocationProposalBundleEvent` to Cache to commit.
+
+Cache look at AllocationProposal under lock, and commit these proposals. The reason to do proposal/commit is Scheduler can run in multi-threads which could cause conflict for resource allocation. This approach is inspired by Borg/Omega/YARN Global Scheduling.
+
+Cache checks more states such as queue resources, node resources (we cannot allocate more resource than nodes' available), etc. Once check is done, Cache updates internal data strcture and send confirmation to Scheduler to update the same, and scheduler sends allocated Allocation to RMProxy so Shim can do further options. For example, K8shim will `bind` an allocation (POD) to kubelet.
+
+```
+Job Add:
+--------
+RM -> Cache -> Scheduler (Implemented)
+
+Job Remove:
+-----------
+RM -> Scheduler -> Cache (Implemented)
+Released allocations: (Same as normal release) (Implemented)
+Note: Make sure remove from scheduler first to avoid new allocated created.
+
+Scheduling Request Add:
+-----------------------
+RM -> Cache -> Scheduler (Implemented)
+Note: Will check if requested job exists, queue exists, etc.
+When any request invalid:
+ Cache -> RM (Implemented)
+ Scheduler -> RM (Implemented)
+
+Scheduling Request remove:
+-------------------------
+RM -> Scheduler -> Cache (Implemented)
+Note: Make sure removed from scheduler first to avoid new container allocated
+
+Allocation remove (Preemption)
+-----------------
+Scheduler -> Cache -> RM (TODO)
+ (confirmation)
+
+Allocation remove (RM voluntarilly ask)
+---------------------------------------
+RM -> Scheduler -> Cache -> RM. (Implemented)
+ (confirmation)
+
+Node Add:
+---------
+RM -> Cache -> Scheduler (Implemented)
+Note: Inside Cache, update allocated resources.
+Error handling: Reject Node to RM (Implemented)
+
+Node Remove:
+------------
+Implemented in cache side
+RM -> Scheduler -> Cache (TODO)
+
+Allocation Proposal:
+--------------------
+Scheduler -> Cache -> RM
+When rejected/accepted:
+ Cache -> Scheduler
+
+Initial: (TODO)
+--------
+1. Admin configured partitions
+2. Cache initializes
+3. Scheduler copies configurations
+
+Relations between Entities
+-------------------------
+1. RM includes one or multiple:
+ - Partitions
+ - Jobs
+ - Nodes
+ - Queues
+
+2. One queue:
+ - Under one partition
+ - Under one RM.
+
+3. One job:
+ - Under one queue (Job with same name can under different partitions)
+ - Under one partition
+
+RM registration: (TODO)
+----------------
+1. RM send registration
+2. If RM already registered, remove old one, including everything belong to RM.
+
+RM termination (TODO)
+--------------
+Just remove the old one.
+
+Update of queues (TODO)
+------------------------
+Admin Service -> Cache
+
+About partition (TODO)
+-----------------------
+Internal partition need to be normalized, for example, RM specify node with partition = xyz.
+Scheduler internally need to normalize it to <rm-id>_xyz
+This need to be done by RMProxy
+
+```
diff --git a/versioned_docs/version-1.5.1/archived_design/state_aware_scheduling.md b/versioned_docs/version-1.5.1/archived_design/state_aware_scheduling.md
new file mode 100644
index 0000000000..f92f93cff4
--- /dev/null
+++ b/versioned_docs/version-1.5.1/archived_design/state_aware_scheduling.md
@@ -0,0 +1,112 @@
+---
+id: state_aware_scheduling
+title: Batch Workloads Ordering with StateAware Policy
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+## The problem
+A common pattern while processing data is that the application can be divided into multiple stages.
+Another way to look at this is the fact that processing needs to be kicked off and that the first step is to start a driver or manager for the application.
+Later stages might depend on the previous stages.
+When running applications in a size limited environment this could lead to a resource exhaustion when submitting multiple applications at the same time.
+These first stages might consume all available resources leaving no room for the next stage(s) to start.
+Often this issue is caused by having a high number of applications start simultaneous and trying to get resources in parallel.
+### Example issue
+When submitting numerous Spark applications in a short amount of time the drivers will all be started shortly after each other.
+The drivers consume all available resources in a queue or in the whole cluster.
+After starting the drivers they will request resources for the executors.
+Since the queue or cluster has no resources left the executors will not be started.
+The driver cannot progress.
+The only way that progress would be made is if and when one of the drivers finishes or fails and frees up resources for executors to be started.
+
+## Design
+### Design goals
+1. Prevent resource exhaustion by first stage allocations
+1. Improve chance for jobs to get minimal required resources over others
+
+### None goals
+1. This is NOT an implementation of Gang scheduling.
+1. No change to the currently supported FAIR or FIFO scheduling algorithms
+1. Fix resource quota usage outside of the core scheduler for submitted but waiting applications
+
+### Possible solutions
+Other batch schedulers like the YARN schedulers use a limit on the number of simultaneous running applications.
+They use either resource constraints on the driver or management stage or set a hard limit of the number of applications that can run in a queue.
+The draw back of that solution is that it does not work well in a cluster that can scale up or down automatically in a cloud environment.
+To work around that percentage based limits could be set on the consumed resources for the driver or management stage.
+This does not alleviate the fact that driver or management stages can be of any size, large and or small, which complicates the percentage scenario further as it does not give a predictable behaviour.
+
+A different solution would be to assume a specific behaviour of the applications.
+Using that assumption a limit on the applications could be set based on the state it is in.
+The spark driver and executor behaviour is the most usual use case.
+This would provide a way to limit scheduling to existing applications and only drip feed new applications into the list of applications to schedule when there are resources available.
+
+### Algorithm
+The algorithm described here is based on the drip feed of new applications into the applications to schedule based on the states of all applications.
+Scheduling is based on the applications in a queue.
+The algorithm will be applied at a queue level.
+This is not a cluster wide setup.
+
+What we want to achieve is the following behaviour: only schedule one (1) application that is in its early stage(s) (called a starting state) at the same time.
+Only consider another new application if and when the previous application has transitioned out of the starting state.
+Applications will always be allocated resources on a first in first out basis based on submission time.
+That means that an application that is newly added and in its starting phase will only get resources if applications in the later stages do not need any resources.
+
+This algorithm will be implemented as an application sorting policy on a queue.
+This allows specific queues to limit parallel application startup while other queues with different work loads can schedule without or with different limitations.
+
+### Fallback mechanism
+A fallback mechanism has to be built into the algorithm.
+Not all applications will request more than one allocation.
+The other case that has to be accounted for could be a misbehaving or a slow application.
+Having an application stuck in the starting state could cause a scheduler livelock and starvation of other applications.
+
+The fall back mechanism proposed is as simple as a time limit on the starting state.
+This means that any application auto progresses out of the starting state.
+The time limit will be set to five (5) minutes hard coded as a starting point and will not be made configurable.
+
+The other fallback mechanism considered was making the number of allocations for the starting state configurable.
+This option provided a number of issues which made it difficult to implement.
+One of the main stumbling blocks is the fact that it requires the application submitter to specify the value.
+It also does not guarantee that the application will leave the starting state either and does not fix the livelock issue.
+If an application was submitted with five required allocation but due to issues during the run never asked for more than four then the livelock would still occur.
+
+Setting a default of zero (0) would also not work as it would bypass the starting state.
+It would make the sorting policy an opt-in instead of an opt-out.
+Setting a default of one (1) does not give us much enhancement to what we currently propose.
+It makes the sorting policy an opt-out but does not give the cluster administrator any control over the scheduling behaviour.
+Weighing those against each other the proposal is to not make this configurable.
+
+### Example run
+Using Spark applications as an example: a new application can only be scheduled if the previous application has at least one (1) executor allocated.
+
+
+
+Assume we have the following Spark apps: App1 & App2 as in the diagram above. The applications were submitted in that order: App1 first, then App2. They were both submitted to the same queue.
+
+1. Both applications are in the queue waiting for the first allocation: accepted by the scheduler. App1 has requested driver D1 and App2 has requested driver D2.
+1. The scheduler sorts the application and allows 1 accepted application to be scheduled (no starting applications yet): App1 as the oldest applications with an outstanding request is scheduled.
+App1 is allocated its driver (D1) and progresses to starting.
+App2 request for a driver is ignored as the scheduler is starting App1 (only 1 application in starting or accepted state is scheduled).
+1. App1 requests executors E11 and E12. The scheduler assigns E11 and E12. At this point the application state changes to running when it has at least 1 executor allocated.
+1. App2 has been waiting to get the driver allocated. Since there are no applications in a starting state the scheduler looks at App2 which is in an accepted state. App2 moves from the accepted state to starting when the driver is allocated.
+1. App2 requests its executor E21. The application state changes to running when E21 is allocated.
+
+This process would repeat itself for any new application submitted.
\ No newline at end of file
diff --git a/versioned_docs/version-1.5.1/assets/RunningSparkOnK8s.png b/versioned_docs/version-1.5.1/assets/RunningSparkOnK8s.png
new file mode 100644
index 0000000000..7594a07585
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/RunningSparkOnK8s.png differ
diff --git a/versioned_docs/version-1.5.1/assets/allocation_4k.png b/versioned_docs/version-1.5.1/assets/allocation_4k.png
new file mode 100644
index 0000000000..03346f52d8
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/allocation_4k.png differ
diff --git a/versioned_docs/version-1.5.1/assets/application-state.png b/versioned_docs/version-1.5.1/assets/application-state.png
new file mode 100644
index 0000000000..ec1b365b95
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/application-state.png differ
diff --git a/versioned_docs/version-1.5.1/assets/architecture.png b/versioned_docs/version-1.5.1/assets/architecture.png
new file mode 100644
index 0000000000..a19dcaa42d
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/architecture.png differ
diff --git a/versioned_docs/version-1.5.1/assets/cpu_profile.jpg b/versioned_docs/version-1.5.1/assets/cpu_profile.jpg
new file mode 100644
index 0000000000..7e99f62306
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/cpu_profile.jpg differ
diff --git a/versioned_docs/version-1.5.1/assets/dashboard_secret.png b/versioned_docs/version-1.5.1/assets/dashboard_secret.png
new file mode 100644
index 0000000000..60b4f976ce
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/dashboard_secret.png differ
diff --git a/versioned_docs/version-1.5.1/assets/dashboard_token_select.png b/versioned_docs/version-1.5.1/assets/dashboard_token_select.png
new file mode 100644
index 0000000000..59173fdab9
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/dashboard_token_select.png differ
diff --git a/versioned_docs/version-1.5.1/assets/docker-dektop-minikube.png b/versioned_docs/version-1.5.1/assets/docker-dektop-minikube.png
new file mode 100644
index 0000000000..48b3584f22
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/docker-dektop-minikube.png differ
diff --git a/versioned_docs/version-1.5.1/assets/docker-desktop.png b/versioned_docs/version-1.5.1/assets/docker-desktop.png
new file mode 100644
index 0000000000..9224360397
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/docker-desktop.png differ
diff --git a/versioned_docs/version-1.5.1/assets/external_userinfo.png b/versioned_docs/version-1.5.1/assets/external_userinfo.png
new file mode 100644
index 0000000000..1598351b8b
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/external_userinfo.png differ
diff --git a/versioned_docs/version-1.5.1/assets/fifo-state-example.png b/versioned_docs/version-1.5.1/assets/fifo-state-example.png
new file mode 100644
index 0000000000..ca04c17d88
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/fifo-state-example.png differ
diff --git a/versioned_docs/version-1.5.1/assets/gang_clean_up.png b/versioned_docs/version-1.5.1/assets/gang_clean_up.png
new file mode 100644
index 0000000000..baf5accb48
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/gang_clean_up.png differ
diff --git a/versioned_docs/version-1.5.1/assets/gang_generic_flow.png b/versioned_docs/version-1.5.1/assets/gang_generic_flow.png
new file mode 100644
index 0000000000..381b7d00c8
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/gang_generic_flow.png differ
diff --git a/versioned_docs/version-1.5.1/assets/gang_scheduling_intro.png b/versioned_docs/version-1.5.1/assets/gang_scheduling_intro.png
new file mode 100644
index 0000000000..b3be207eab
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/gang_scheduling_intro.png differ
diff --git a/versioned_docs/version-1.5.1/assets/gang_timeout.png b/versioned_docs/version-1.5.1/assets/gang_timeout.png
new file mode 100644
index 0000000000..a8ea0daf1c
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/gang_timeout.png differ
diff --git a/versioned_docs/version-1.5.1/assets/gang_total_ask.png b/versioned_docs/version-1.5.1/assets/gang_total_ask.png
new file mode 100644
index 0000000000..928d7fdf56
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/gang_total_ask.png differ
diff --git a/versioned_docs/version-1.5.1/assets/goland_debug.jpg b/versioned_docs/version-1.5.1/assets/goland_debug.jpg
new file mode 100644
index 0000000000..c9ab94ca0d
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/goland_debug.jpg differ
diff --git a/versioned_docs/version-1.5.1/assets/goland_debug.png b/versioned_docs/version-1.5.1/assets/goland_debug.png
new file mode 100644
index 0000000000..aa8803f804
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/goland_debug.png differ
diff --git a/versioned_docs/version-1.5.1/assets/grafana_dashboard.png b/versioned_docs/version-1.5.1/assets/grafana_dashboard.png
new file mode 100644
index 0000000000..b097ee3769
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/grafana_dashboard.png differ
diff --git a/versioned_docs/version-1.5.1/assets/grafana_login_page.png b/versioned_docs/version-1.5.1/assets/grafana_login_page.png
new file mode 100644
index 0000000000..9aa818b4c9
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/grafana_login_page.png differ
diff --git a/versioned_docs/version-1.5.1/assets/import_dashboard_01.png b/versioned_docs/version-1.5.1/assets/import_dashboard_01.png
new file mode 100644
index 0000000000..60d51d86b9
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/import_dashboard_01.png differ
diff --git a/versioned_docs/version-1.5.1/assets/import_dashboard_02.png b/versioned_docs/version-1.5.1/assets/import_dashboard_02.png
new file mode 100644
index 0000000000..bbf18ab549
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/import_dashboard_02.png differ
diff --git a/versioned_docs/version-1.5.1/assets/k8shim-application-state.png b/versioned_docs/version-1.5.1/assets/k8shim-application-state.png
new file mode 100644
index 0000000000..c286591964
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/k8shim-application-state.png differ
diff --git a/versioned_docs/version-1.5.1/assets/k8shim-task-state.png b/versioned_docs/version-1.5.1/assets/k8shim-task-state.png
new file mode 100644
index 0000000000..11c95d1317
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/k8shim-task-state.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-affinity.png b/versioned_docs/version-1.5.1/assets/kwok-affinity.png
new file mode 100644
index 0000000000..a79d93592c
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-affinity.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-priorityclass.png b/versioned_docs/version-1.5.1/assets/kwok-priorityclass.png
new file mode 100644
index 0000000000..e704409d8c
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-priorityclass.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-throughput-01.png b/versioned_docs/version-1.5.1/assets/kwok-throughput-01.png
new file mode 100644
index 0000000000..59e34a5925
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-throughput-01.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-throughput-02.png b/versioned_docs/version-1.5.1/assets/kwok-throughput-02.png
new file mode 100644
index 0000000000..3e186a9b6e
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-throughput-02.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-throughput-03.png b/versioned_docs/version-1.5.1/assets/kwok-throughput-03.png
new file mode 100644
index 0000000000..6b87e3243b
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-throughput-03.png differ
diff --git a/versioned_docs/version-1.5.1/assets/kwok-toleration-taint.png b/versioned_docs/version-1.5.1/assets/kwok-toleration-taint.png
new file mode 100644
index 0000000000..183c721653
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/kwok-toleration-taint.png differ
diff --git a/versioned_docs/version-1.5.1/assets/namespace-mapping.png b/versioned_docs/version-1.5.1/assets/namespace-mapping.png
new file mode 100644
index 0000000000..9ad07da8f1
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/namespace-mapping.png differ
diff --git a/versioned_docs/version-1.5.1/assets/node-bin-packing.png b/versioned_docs/version-1.5.1/assets/node-bin-packing.png
new file mode 100644
index 0000000000..9267a03596
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/node-bin-packing.png differ
diff --git a/versioned_docs/version-1.5.1/assets/node-fair.png b/versioned_docs/version-1.5.1/assets/node-fair.png
new file mode 100644
index 0000000000..2e404abbc9
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/node-fair.png differ
diff --git a/versioned_docs/version-1.5.1/assets/node_fairness_conf.png b/versioned_docs/version-1.5.1/assets/node_fairness_conf.png
new file mode 100644
index 0000000000..80e1dd5768
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/node_fairness_conf.png differ
diff --git a/versioned_docs/version-1.5.1/assets/object-state.png b/versioned_docs/version-1.5.1/assets/object-state.png
new file mode 100644
index 0000000000..9baca07cef
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/object-state.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf-tutorial-build.png b/versioned_docs/version-1.5.1/assets/perf-tutorial-build.png
new file mode 100644
index 0000000000..5c2f28bd07
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf-tutorial-build.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf-tutorial-resultDiagrams.png b/versioned_docs/version-1.5.1/assets/perf-tutorial-resultDiagrams.png
new file mode 100644
index 0000000000..6d9686d48d
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf-tutorial-resultDiagrams.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf-tutorial-resultLog.png b/versioned_docs/version-1.5.1/assets/perf-tutorial-resultLog.png
new file mode 100755
index 0000000000..dad58bf14e
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf-tutorial-resultLog.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf_e2e_test.png b/versioned_docs/version-1.5.1/assets/perf_e2e_test.png
new file mode 100644
index 0000000000..79763d03ea
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf_e2e_test.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf_e2e_test_conf.png b/versioned_docs/version-1.5.1/assets/perf_e2e_test_conf.png
new file mode 100644
index 0000000000..5aeef943cd
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf_e2e_test_conf.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf_node_fairness.png b/versioned_docs/version-1.5.1/assets/perf_node_fairness.png
new file mode 100644
index 0000000000..fbacc13bef
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf_node_fairness.png differ
diff --git a/versioned_docs/version-1.5.1/assets/perf_throughput.png b/versioned_docs/version-1.5.1/assets/perf_throughput.png
new file mode 100644
index 0000000000..e89a7f3888
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/perf_throughput.png differ
diff --git a/versioned_docs/version-1.5.1/assets/pluggable-app-mgmt.jpg b/versioned_docs/version-1.5.1/assets/pluggable-app-mgmt.jpg
new file mode 100644
index 0000000000..443b8ad336
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/pluggable-app-mgmt.jpg differ
diff --git a/versioned_docs/version-1.5.1/assets/predicateComaparation.png b/versioned_docs/version-1.5.1/assets/predicateComaparation.png
new file mode 100755
index 0000000000..d3498c8bd4
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/predicateComaparation.png differ
diff --git a/versioned_docs/version-1.5.1/assets/predicate_4k.png b/versioned_docs/version-1.5.1/assets/predicate_4k.png
new file mode 100644
index 0000000000..850036c267
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/predicate_4k.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_fence.png b/versioned_docs/version-1.5.1/assets/preemption_fence.png
new file mode 100644
index 0000000000..61222bcd89
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_fence.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_fence_case1.png b/versioned_docs/version-1.5.1/assets/preemption_fence_case1.png
new file mode 100644
index 0000000000..cb8f7b47d1
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_fence_case1.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_fence_case2.png b/versioned_docs/version-1.5.1/assets/preemption_fence_case2.png
new file mode 100644
index 0000000000..6156ebf403
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_fence_case2.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_fence_hierarchy_case.png b/versioned_docs/version-1.5.1/assets/preemption_fence_hierarchy_case.png
new file mode 100644
index 0000000000..26db9c76d2
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_fence_hierarchy_case.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_normal_case.png b/versioned_docs/version-1.5.1/assets/preemption_normal_case.png
new file mode 100644
index 0000000000..18644222da
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_normal_case.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_priority_queue_case.png b/versioned_docs/version-1.5.1/assets/preemption_priority_queue_case.png
new file mode 100644
index 0000000000..6d6fb131be
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_priority_queue_case.png differ
diff --git a/versioned_docs/version-1.5.1/assets/preemption_priorityclass_case.png b/versioned_docs/version-1.5.1/assets/preemption_priorityclass_case.png
new file mode 100644
index 0000000000..91676b07ce
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/preemption_priorityclass_case.png differ
diff --git a/versioned_docs/version-1.5.1/assets/priority-flows.png b/versioned_docs/version-1.5.1/assets/priority-flows.png
new file mode 100644
index 0000000000..19b82ec062
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/priority-flows.png differ
diff --git a/versioned_docs/version-1.5.1/assets/priority-tree.png b/versioned_docs/version-1.5.1/assets/priority-tree.png
new file mode 100644
index 0000000000..7288db5ea5
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/priority-tree.png differ
diff --git a/versioned_docs/version-1.5.1/assets/prometheus-cmd.png b/versioned_docs/version-1.5.1/assets/prometheus-cmd.png
new file mode 100644
index 0000000000..86bfc6a494
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/prometheus-cmd.png differ
diff --git a/versioned_docs/version-1.5.1/assets/prometheus-web-ui.png b/versioned_docs/version-1.5.1/assets/prometheus-web-ui.png
new file mode 100644
index 0000000000..b4519765ac
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/prometheus-web-ui.png differ
diff --git a/versioned_docs/version-1.5.1/assets/prometheus.png b/versioned_docs/version-1.5.1/assets/prometheus.png
new file mode 100644
index 0000000000..964c79ca08
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/prometheus.png differ
diff --git a/versioned_docs/version-1.5.1/assets/queue-fairness.png b/versioned_docs/version-1.5.1/assets/queue-fairness.png
new file mode 100644
index 0000000000..7c78ed70b1
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/queue-fairness.png differ
diff --git a/versioned_docs/version-1.5.1/assets/queue-resource-quotas.png b/versioned_docs/version-1.5.1/assets/queue-resource-quotas.png
new file mode 100644
index 0000000000..fb72138941
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/queue-resource-quotas.png differ
diff --git a/versioned_docs/version-1.5.1/assets/quota_check.png b/versioned_docs/version-1.5.1/assets/quota_check.png
new file mode 100644
index 0000000000..b8d29e6c2c
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/quota_check.png differ
diff --git a/versioned_docs/version-1.5.1/assets/resilience-node-recovery.jpg b/versioned_docs/version-1.5.1/assets/resilience-node-recovery.jpg
new file mode 100644
index 0000000000..384745164f
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/resilience-node-recovery.jpg differ
diff --git a/versioned_docs/version-1.5.1/assets/resilience-workflow.jpg b/versioned_docs/version-1.5.1/assets/resilience-workflow.jpg
new file mode 100644
index 0000000000..40ab6baf24
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/resilience-workflow.jpg differ
diff --git a/versioned_docs/version-1.5.1/assets/scheduling_no_predicate_4k.png b/versioned_docs/version-1.5.1/assets/scheduling_no_predicate_4k.png
new file mode 100644
index 0000000000..0ebe41c813
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/scheduling_no_predicate_4k.png differ
diff --git a/versioned_docs/version-1.5.1/assets/scheduling_with_predicate_4k_.png b/versioned_docs/version-1.5.1/assets/scheduling_with_predicate_4k_.png
new file mode 100644
index 0000000000..2cee7c0142
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/scheduling_with_predicate_4k_.png differ
diff --git a/versioned_docs/version-1.5.1/assets/simple_preemptor.png b/versioned_docs/version-1.5.1/assets/simple_preemptor.png
new file mode 100644
index 0000000000..c5165c341e
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/simple_preemptor.png differ
diff --git a/versioned_docs/version-1.5.1/assets/sparkResult.png b/versioned_docs/version-1.5.1/assets/sparkResult.png
new file mode 100644
index 0000000000..8a11e9b721
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/sparkResult.png differ
diff --git a/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-logs.png b/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-logs.png
new file mode 100644
index 0000000000..db2a6b693b
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-logs.png differ
diff --git a/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-ui.png b/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-ui.png
new file mode 100644
index 0000000000..b599dca7b8
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/tf-job-gpu-on-ui.png differ
diff --git a/versioned_docs/version-1.5.1/assets/tf-job-on-ui.png b/versioned_docs/version-1.5.1/assets/tf-job-on-ui.png
new file mode 100644
index 0000000000..06acabec29
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/tf-job-on-ui.png differ
diff --git a/versioned_docs/version-1.5.1/assets/throughput.png b/versioned_docs/version-1.5.1/assets/throughput.png
new file mode 100644
index 0000000000..8ced22c815
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/throughput.png differ
diff --git a/versioned_docs/version-1.5.1/assets/throughput_3types.png b/versioned_docs/version-1.5.1/assets/throughput_3types.png
new file mode 100644
index 0000000000..a4a583b518
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/throughput_3types.png differ
diff --git a/versioned_docs/version-1.5.1/assets/throughput_conf.png b/versioned_docs/version-1.5.1/assets/throughput_conf.png
new file mode 100644
index 0000000000..dd5d720634
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/throughput_conf.png differ
diff --git a/versioned_docs/version-1.5.1/assets/translationBuilding.png b/versioned_docs/version-1.5.1/assets/translationBuilding.png
new file mode 100644
index 0000000000..0ba725051c
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/translationBuilding.png differ
diff --git a/versioned_docs/version-1.5.1/assets/translationDemo.png b/versioned_docs/version-1.5.1/assets/translationDemo.png
new file mode 100644
index 0000000000..4125dc676f
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/translationDemo.png differ
diff --git a/versioned_docs/version-1.5.1/assets/ug_lookup.png b/versioned_docs/version-1.5.1/assets/ug_lookup.png
new file mode 100644
index 0000000000..d86ae6a1ae
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/ug_lookup.png differ
diff --git a/versioned_docs/version-1.5.1/assets/yk-ui-screenshots.gif b/versioned_docs/version-1.5.1/assets/yk-ui-screenshots.gif
new file mode 100644
index 0000000000..5ce27e1688
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/yk-ui-screenshots.gif differ
diff --git a/versioned_docs/version-1.5.1/assets/yunikorn-gpu-time-slicing.png b/versioned_docs/version-1.5.1/assets/yunikorn-gpu-time-slicing.png
new file mode 100644
index 0000000000..8b3d734a4d
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/yunikorn-gpu-time-slicing.png differ
diff --git a/versioned_docs/version-1.5.1/assets/yunirkonVSdefault.png b/versioned_docs/version-1.5.1/assets/yunirkonVSdefault.png
new file mode 100755
index 0000000000..a123ff8db2
Binary files /dev/null and b/versioned_docs/version-1.5.1/assets/yunirkonVSdefault.png differ
diff --git a/versioned_docs/version-1.5.1/design/architecture.md b/versioned_docs/version-1.5.1/design/architecture.md
new file mode 100644
index 0000000000..5163d4c9fb
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/architecture.md
@@ -0,0 +1,62 @@
+---
+id: architecture
+title: Architecture
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Apache YuniKorn is a light-weight, universal resource scheduler for container orchestrator systems.
+It is created to achieve fine-grained resource sharing for various workloads efficiently on a large scale, multi-tenant,
+and cloud-native environment. YuniKorn brings a unified, cross-platform, scheduling experience for mixed workloads that
+consist of stateless batch workloads and stateful services.
+
+YuniKorn now supports K8s and can be deployed as a custom K8s scheduler. YuniKorn's architecture design also allows
+adding different shim layer and adopt to different ResourceManager implementation including Apache Hadoop YARN,
+or any other systems.
+
+## Architecture
+
+Following chart illustrates the high-level architecture of YuniKorn.
+
+<img src={require('./../assets/architecture.png').default} />
+
+## Components
+
+### Scheduler interface
+
+[Scheduler interface](https://github.com/apache/yunikorn-scheduler-interface) is an abstract layer
+which resource management platform (like YARN/K8s) will speak with, via API like GRPC/programing language bindings.
+
+### Scheduler core
+
+Scheduler core encapsulates all scheduling algorithms, it collects resources from underneath resource management
+platforms (like YARN/K8s), and is responsible for container allocation requests. It makes the decision where is the
+best spot for each request and then sends response allocations to the resource management platform.
+Scheduler core is agnostic about underneath platforms, all the communications are through the [scheduler interface](https://github.com/apache/yunikorn-scheduler-interface).
+Please read more about the design of schedule core [here](../archived_design/scheduler_core_design.md).
+
+### Kubernetes shim
+
+The YuniKorn Kubernetes shim is responsible for talking to Kubernetes, it is responsible for translating the Kubernetes
+cluster resources, and resource requests via scheduler interface and send them to the scheduler core.
+And when a scheduler decision is made, it is responsible for binding the pod to the specific node. All the communication
+between the shim and the scheduler core is through the [scheduler interface](https://github.com/apache/yunikorn-scheduler-interface).
+Please read more about the design of the Kubernetes shim [here](../archived_design/k8shim.md).
+
diff --git a/versioned_docs/version-1.5.1/design/cache_removal.md b/versioned_docs/version-1.5.1/design/cache_removal.md
new file mode 100644
index 0000000000..2312643305
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/cache_removal.md
@@ -0,0 +1,456 @@
+---
+id: cache_removal
+title: Scheduler cache removal design
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+:::caution
+The Interface Message definitions described in this design doc has undergone major refactoring to reduce the complexity. [YUNIKORN-337](https://issues.apache.org/jira/browse/YUNIKORN-337) was committed and simplified the message communication between Core and Shim to greater extent.
+See [Simplifying Interface Messages and Breaking Shim build dependency on Core](interface_message_simplification.md) to know the updated message definitions.
+:::
+
+# Proposal to combine Cache and Scheduler's implementation in the core
+This document describes the current state of the scheduler and cache implementation.
+It describes the changes planned based on the analysis that was done of the current behaviour.
+
+## Goals
+The goal is to provide the same functionality before and after the change.
+- Unit tests before and after the merge must all pass.
+- Smoke tests defined in the core should all pass without major changes.
+- End-to-end tests that are part of the shim code must all pass without changes.
+
+:::info
+ Major changes for smoke tests are defined as changes to the tests that alter use case and thus test flows. Some changes will be needed as checks made could rely on cache objects which have been removed.
+:::
+
+## Background
+The current Scheduler Core is build up around two major components to store the data: the cache and scheduler objects.
+The cache objects form the base for most data to be tracked.
+The Scheduler objects track specific in flight details and are build on top of a cache object.
+
+The communication between the two layers uses a-synchronous events and in some cases direct updates.
+A synchronous update between the scheduler and the cache does mean that there is a short period the scheduler is "out of sync" with the cache.
+This short period can have an impact on the scheduling decisions.
+One of which is logged as [YUNIKORN-169](https://issues.apache.org/jira/browse/YUNIKORN-169).
+
+A further point is the complexity that the two structure brings to the code.
+A distinct set of messages to communicate between the scheduler and the cache. A one-on-one mapping between the scheduler and cache objects shows that the distinction is probably more artificial than required.
+
+
+---
+
+## Structure analysis
+### Objects
+The existing objects as per the code analysis.
+The overlap between the scheduler and the cache objects is shown by showing them at the same line.
+N/A means that there is no equivalent object in either the scheduler or cache.
+
+| Cache Object | Scheduler Object |
+| ------------------------------ | ------------------------------ |
+| ClusterInfo | ClusterSchedulingContext |
+| PartitionInfo | partitionSchedulingContext |
+| AllocationInfo | schedulingAllocation |
+| N/A | schedulingAllocationAsk |
+| N/A | reservation |
+| ApplicationInfo | SchedulingApplication |
+| applicationState | N/A |
+| NodeInfo | SchedulingNode |
+| QueueInfo | SchedulingQueue |
+| SchedulingObjectState | N/A |
+
+The `initializer` code that is part of the cache does not define a specific object.
+It contains a mixture of code defined at the package level and code that is part of the `ClusterInfo` object.
+
+### Events
+Events defined in the core have multiple origins and destinations.
+Some events are only internal for the core between the cache and scheduler.
+These events will be removed.
+
+| Event | Flow | Proposal |
+| ----------------------------------------- | --------------------- | -------- |
+| AllocationProposalBundleEvent | Scheduler -> Cache | Remove |
+| RejectedNewApplicationEvent | Scheduler -> Cache | Remove |
+| ReleaseAllocationsEvent | Scheduler -> Cache | Remove |
+| RemoveRMPartitionsEvent | Scheduler -> Cache | Remove |
+| RemovedApplicationEvent | Scheduler -> Cache | Remove |
+| SchedulerNodeEvent | Cache -> Scheduler | Remove |
+| SchedulerAllocationUpdatesEvent | Cache -> Scheduler | Remove |
+| SchedulerApplicationsUpdateEvent | Cache -> Scheduler | Remove |
+| SchedulerUpdatePartitionsConfigEvent | Cache -> Scheduler | Remove |
+| SchedulerDeletePartitionsConfigEvent | Cache -> Scheduler | Remove |
+| RMApplicationUpdateEvent (add/remove app) | Cache/Scheduler -> RM | Modify |
+| RMRejectedAllocationAskEvent | Cache/Scheduler -> RM | Modify |
+| RemoveRMPartitionsEvent | RM -> Scheduler | |
+| RMUpdateRequestEvent | RM -> Cache | Modify |
+| RegisterRMEvent | RM -> Cache | Modify |
+| ConfigUpdateRMEvent | RM -> Cache | Modify |
+| RMNewAllocationsEvent | Cache -> RM | Modify |
+| RMReleaseAllocationEvent | Cache -> RM | Modify |
+| RMNodeUpdateEvent | Cache -> RM | Modify |
+| | | |
+
+Events that are handled by the cache will need to be handled by the core code after the removal of the cache.
+Two events are handled by the cache and the scheduler.
+
+## Detailed flow analysis
+### Object existing in both cache and scheduler
+The current design is based on the fact that the cache object is the basis for all data storage.
+Each cache object must have a corresponding scheduler object.
+The contract in the core around the cache and scheduler objects was simple.
+If the object exists in both scheduler and cache the object will be added to cache triggering the creation of the corresponding scheduler object.
+Removing the object is always handled in reverse: first from the scheduler which will trigger the removal from the cache.
+An example would be the creation of an application triggered by the `RMUpdateRequestEvent` would be processed by the cache.
+Creating a `SchedulerApplicationsUpdateEvent` to create the corresponding application in the scheduler.
+
+When the application and object state were added they were added into the cache objects.
+The cache objects were considered the data store and thus also contain the state.
+There were no corresponding state objects in the scheduler.
+Maintaining two states for the same object is not possible.
+
+The other exceptions to that rule are two objects that were considered volatile and scheduler only.
+The `schedulingAllocationAsk` tracks outstanding requests for an application in the scheduler.
+The `reservation` tracks a temporary reservation of a node for an application and ask combination.
+
+### Operations to add/remove app
+The RM (shim) sends a complex `UpdateRequest` as defined in the scheduler interface.
+This message is wrapped by the RM proxy and forwarded to the cache for processing.
+The RM can request an application to be added or removed.
+
+**application add or delete**
+```
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processApplicationUpdateFromRMUpdate
+ 2.1: Add new apps to the partition.
+ 2.2: Send removed apps to scheduler (but not remove anything from cache)
+3. scheduler.processApplicationUpdateEvent
+ 3.1: Add new apps to scheduler
+ (when fails, send RejectedNewApplicationEvent to cache)
+ No matter if failed or not, send RMApplicationUpdateEvent to RM.
+ 3.2: Remove app from scheduler
+ Send RemovedApplicationEvent to cache
+```
+
+### Operations to remove allocations and add or remove asks
+The RM (shim) sends a complex `UpdateRequest` as defined in the scheduler interface.
+This message is wrapped by the RM proxy and forwarded to the cache for processing.
+The RM can request an allocation to be removed.
+The RM can request an ask to be added or removed
+
+**allocation delete**
+This describes the allocation delete initiated by the RM only
+````
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processNewAndReleaseAllocationRequests
+ 2.1: (by-pass): Send to scheduler via event SchedulerAllocationUpdatesEvent
+3. scheduler.processAllocationUpdateEvent
+ 3.1: Update ReconcilePlugin
+ 3.2: Send confirmation of the releases back to Cache via event ReleaseAllocationsEvent
+4. cluster_info.processAllocationReleases to process the confirmed release
+````
+
+**ask add**
+If the ask already exists this add is automatically converted into an update.
+```
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processNewAndReleaseAllocationRequests
+ 2.1: Ask sanity check (such as existence of partition/app), rejections are send back to the RM via RMRejectedAllocationAskEvent
+ 2.2: pass checked asks to scheduler via SchedulerAllocationUpdatesEvent
+3. scheduler.processAllocationUpdateEvent
+ 3.1: Update scheduling application with the new or updated ask.
+ 3.2: rejections are send back to the RM via RMRejectedAllocationAskEvent
+ 3.3: accepted asks are not confirmed to RM or cache
+```
+
+**ask delete**
+```
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processNewAndReleaseAllocationRequests
+ 2.1: (by-pass): Send to scheduler via event SchedulerAllocationUpdatesEvent
+3. scheduler.processAllocationReleaseByAllocationKey
+ 3.1: Update scheduling application and remove the ask.
+```
+
+### Operations to add, update or remove nodes
+The RM (shim) sends a complex `UpdateRequest` as defined in the scheduler interface.
+This message is wrapped by the RM proxy and forwarded to the cache for processing.
+The RM can request a node to be added, updated or removed.
+
+**node add**
+```
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processNewSchedulableNodes
+ 2.1: node sanity check (such as existence of partition/node)
+ 2.2: Add new nodes to the partition.
+ 2.3: notify scheduler of new node via SchedulerNodeEvent
+3. notify RM of node additions and rejections via RMNodeUpdateEvent
+ 3.1: notify the scheduler of allocations to recover via SchedulerAllocationUpdatesEvent
+4. scheduler.processAllocationUpdateEvent
+ 4.1: scheduler creates a new ask based on the Allocation to recover
+ 4.2: recover the allocation on the new node using a special process
+ 4.3: confirm the allocation in the scheduler, on failure update the cache with a ReleaseAllocationsEvent
+```
+
+**node update and removal**
+```
+1. RMProxy sends cacheevent.RMUpdateRequestEvent to cache
+2. cluster_info.processNodeActions
+ 2.1: node sanity check (such as existence of partition/node)
+ 2.2: Node info update (resource change)
+ 2.2.1: update node in cache
+ 2.2.2: notify scheduler of the node update via SchedulerNodeEvent
+ 2.3: Node status update (not removal), update node status in cache only
+ 2.4: Node removal
+ 2.4.1: update node status and remove node from the cache
+ 2.4.2: remove alloations and inform RM via RMReleaseAllocationEvent
+ 2.4.3: notify scheduler of the node removal via SchedulerNodeEvent
+3. scheduler.processNodeEvent add/remove/update the node
+```
+
+### Operations to add, update or remove partitions
+**Add RM**
+```
+1. RMProxy sends commonevents.RemoveRMPartitionsEvent
+ if RM is already registered
+ 1.1: scheduler.removePartitionsBelongToRM
+ 1.1.1: scheduler cleans up
+ 1.1.2: scheduler sends commonevents.RemoveRMPartitionsEvent
+ 1.2: cluster_info.processRemoveRMPartitionsEvent
+ 1.2.1: cache cleans up
+2. RMProxy sends commonevents.RegisterRMEvent
+3. cluster_info.processRMRegistrationEvent
+ 2.1: cache update internal partitions/queues accordingly.
+ 2.2: cache sends to scheduler SchedulerUpdatePartitionsConfigEvent.
+3. scheduler.processUpdatePartitionConfigsEvent
+ 3.1: Scheduler update partition/queue info accordingly.
+```
+
+**Update and Remove partition**
+Triggered by a configuration file update.
+```
+1. RMProxy sends commonevents.ConfigUpdateRMEvent
+2. cluster_info.processRMConfigUpdateEvent
+ 2.1: cache update internal partitions/queues accordingly.
+ 2.2: cache sends to scheduler SchedulerUpdatePartitionsConfigEvent.
+ 2.3: cache marks partitions for deletion (not removed yet).
+ 2.4: cache sends to scheduler SchedulerDeletePartitionsConfigEvent
+3. scheduler.processUpdatePartitionConfigsEvent
+ 3.1: scheduler updates internal partitions/queues accordingly.
+4. scheduler.processDeletePartitionConfigsEvent
+ 4.1: Scheduler set partitionManager.stop = true.
+ 4.2: PartitionManager removes queues, applications, nodes async.
+ This is the REAL CLEANUP including the cache
+```
+
+### Allocations
+Allocations are initiated by the scheduling process.
+The scheduler creates a SchedulingAllocation on the scheduler side which then gets wrapped in an AllocationProposal.
+The scheduler has checked resources etc already and marked the allocation as inflight.
+This description picks up at the point the allocation will be confirmed and finalised.
+
+**New allocation**
+```
+1. Scheduler wraps an SchedulingAllocation in an AllocationProposalBundleEvent
+2. cluster_info.processAllocationProposalEvent
+ preemption case: release preempted allocations
+ 2.1: release the allocation in the cache
+ 2.2: inform the scheduler the allocation is released via SchedulerNodeEvent
+ 2.3: inform the RM the allocation is released via RMReleaseAllocationEvent
+ all cases: add the new allocation
+ 2.4: add the new allocation to the cache
+ 2.5: rejections are send back to the scheduler via SchedulerAllocationUpdatesEvent
+ 2.6: inform the scheduler the allocation is added via SchedulerAllocationUpdatesEvent
+ 2.7: inform the RM the allocation is added via RMNewAllocationsEvent
+3. scheduler.processAllocationUpdateEvent
+ 3.1: confirmations are added to the scheduler and change from inflight to confirmed.
+ On failure of processing a ReleaseAllocationsEvent is send to the cache *again* to clean up.
+ This is part of the issue in [YUNIKORN-169]
+ cluster_info.processAllocationReleases
+ 3.2: rejections remove the inflight allocation from the scheduler.
+```
+
+## Current locking
+**Cluster Lock:**
+A cluster contains one or more Partition objects. A partition is a sub object of Cluster.
+Adding or Removing ANY Partition requires a write-lock of the cluster.
+Retrieving any object within the cluster will require iterating over the Partition list and thus a read-lock of the cluster
+
+**Partition Lock:**
+The partition object contains all links to Queue, Application or Node objects.
+Adding or Removing ANY Queue, Application or Node needs a write-lock of the partition.
+Retrieving any object within the partition will require a read-lock of the partition to prevent data races
+
+Examples of operation needing a write-lock
+- Allocation processing after scheduling, will change application, queue and node objects.
+ Partition lock is required due to possible updates to reservations.
+- Update of Node Resource
+ It not only affect node's available resource, it also affects the Partition's total allocatable Resource
+
+Example of operations that need a read-lock:
+- Retrieving any Queue, Application or Node needs a read-lock
+ The object itself is not locked as part of the retrieval
+- Confirming an allocation after processing in the cache
+ The partition is only locked for reading to allow retrieval of the objects that will be changed.
+ The changes are made on the underlying objects.
+
+Example of operations that do not need any lock:
+- Scheduling
+ Locks are taken on the specific objects when needed, no direct updates to the partition until the allocation is confirmed.
+
+**Queue lock:**
+A queue can track either applications (leaf type) or other queues (parent type).
+Resources are tracked for both types in the same way.
+
+Adding or removing an Application (leaf type), or a direct child queue (parent type) requires a write-lock of the queue.
+Updating tracked resources requires a write-lock.
+Changes are made recursively never locking more than 1 queue at a time.
+Updating any configuration property on the queue requires a write-lock.
+Retrieving any configuration value, or tracked resource, application or queue requires a read-lock.
+
+Examples of operation needing a write-lock
+- Adding an application to a leaf queue
+- Updating the reservations
+
+Examples of operation needing a read-lock
+- Retrieving an application from a leaf type queue
+- Retrieving the pending resources
+
+**Application lock:**
+An application tracks resources of different types, the allocations and outstanding requests.
+Updating any tracked resources, allocations or requests requires a write-lock.
+Retrieving any of those values requires a read-lock.
+
+Scheduling also requires a write-lock of the application.
+During scheduling the write-lock is held for the application.
+Locks will be taken on the node or queue that need to be accessed or updated.
+Examples of the locks taken on other objects are:
+- a read lock to access queue tracked resources
+- a write-lock to update the in progress allocations on the node
+
+Examples of operation needing a write-lock
+- Adding a new ask
+- Trying to schedule a pending request
+
+Examples of operation needing a read-lock
+- Retrieving the allocated resources
+- Retrieving the pending requests
+
+**Node lock:**
+An node tracks resources of different types and allocations.
+Updating any tracked resources or allocations requires a write-lock.
+Retrieving any of those values requires a read-lock.
+
+Checks run during the allocation phases take locks as required.
+Read-locks when checking write-locks when updating.
+A node is not locked for the whole allocation cycle.
+
+Examples of operation needing a write-lock
+- Adding a new allocation
+- updating the node resources
+
+Examples of operation needing a read-lock
+- Retrieving the allocated resources
+- Retrieving the reservation status
+
+## How to merge Cache and scheduler objects
+Since there is no longer the requirement to distinguish the objects in the cache and scheduler the `scheduling` and `info` parts of the name will be dropped.
+
+Overview of the main moves and merges:
+1. `application_info` & `scheduling_application`: **merge** to `scheduler.object.application`
+2. `allocation_info` & `scheduling_allocation`: **merge** to `scheduler.object.allocation`
+3. `node_info` & `scheduling_node`: **merge** to `scheduler.object.node`
+4. `queue_info` & `scheduling_queue`: **merge** to `scheduler.object.queue`
+5. `partition_info` & `scheduling_partition`: **merge** to `scheduler.PartitionContext`
+6. `cluster_info` & `scheduling_context`: **merge** to `scheduler.ClusterContext`
+7. `application_state`: **move** to `scheduler.object.applicationState`
+8. `object_state`: **move** to `scheduler.object.objectState`
+9. `initializer`: **merge** into `scheduler.ClusterContext`
+
+This move and merge of code includes a refactor of the objects into their own package.
+That thus affects the two scheduler only objects, reservations and schedulingAsk, that are already defined.
+Both will be moved into the objects package.
+
+The top level scheduler package remains for the contexts and scheduling code.
+
+## Code merges
+The first change is the event processing.
+All RM events will now directly be handled in the scheduler.
+Event handling will undergo a major change, far more than a simple merge.
+Only the RM generated events will be left after the merge.
+As described in the analysis above the scheduler is, in almost all cases, notified of changes from RM events.
+
+Broadly speaking there are only three types of changes triggered by the event removal:
+- configuration changes: new scheduler code required as the cache handling is not transferable to the scheduler
+- node, ask and application changes: merge of the cache code into the scheduler
+- allocation changes: removal of confirmation cycle and simplification of the scheduler code
+
+Part of the event handling is the processing of the configuration changes.
+All configuration changes will now update the scheduler objects directly.
+The way the scheduler works is slightly different from the cache which means the code is not transferable.
+
+Nodes and applications are really split between the cache and scheduler.
+Anything that is tracked in the cache object that does not have an equivalent value in the scheduler object will be moved into the scheduler object.
+All references to scheduler objects will be removed.
+With the code merges existing scheduler code that calls out directly into the cache objects will return the newly tracked value in the scheduler object.
+These calls will thus become locked calls in the scheduler.
+
+The concept of an in flight allocation will be removed.
+Allocation will be made in the same scheduling iteration without events or creation of a proposal.
+Removing the need for tracking of allocating resources on the scheduler objects.
+In flight resource tracking was required to make sure that an allocation while not confirmed by the cache would being taken into account while making scheduling decisions.
+
+The application and object state will be an integrated part of the scheduler object.
+A state change is thus immediate and this should prevent an issue like [YUNIKORN-169](https://issues.apache.org/jira/browse/YUNIKORN-169) from occuring.
+
+## Locking after merge
+
+### Direction of lock
+It is possible to acquire another lock while holding a lock, but we need to make sure that we do not allow:
+- Holding A.lock and acquire B's lock.
+- Holding B.lock and acquire B's lock.
+
+The current code in the scheduler takes a lock as late as possible and only for the time period needed.
+Some actions are not locked on the scheduler side just on the cache side as each object has its own lock.
+This means that a read of a value from the cache would not lock the scheduling object.
+
+With the integration of the cache into the scheduler the number of locks will decrease as the number of objects decreases.
+Each equivalent object, cache and scheduler, which used to have their own lock will now have just one.
+After the merge of the code is performed one lock will be left.
+Locking will occur more frequently as the number of fields in the scheduler objects has increased.
+
+Calls that did not lock the scheduler object before the merge will become locked.
+Lock contention could lead to performance degradation.
+The reduced overhead in objects and event handling can hopefully compensate for this.
+One point to keep track of is the change in locking behaviour.
+New behaviour could lead to new deadlock situations when code is simply merged without looking at the order.
+
+### Mitigations for deadlocks
+The locking inside the scheduler will be left as is.
+This means that the main scheduling logic will be taking and releasing locks as required on the objects.
+There are no long held read-locks or write-locks until the application is locked to schedule it.
+
+A major point of attention will need to be that no iterations of objects should be performed while holding on to a lock.
+For instance during scheduling while iterating over a queue's application we should not lock the queue.
+
+Another example would be that event processing in the partition should not lock the partition unneeded.
+The partition should be locked while retrieving for instance the node that needs updating and release the lock before it tries to lock the node itself.
+
+This approach fits in with the current locking approach and will keep the locking changes to a minimum.
+Testing, specifically end-to-end testing, should catch these deadlocks.
+There are no known tools that could be used to detect or describe lock order.
diff --git a/versioned_docs/version-1.5.1/design/config_v2.md b/versioned_docs/version-1.5.1/design/config_v2.md
new file mode 100644
index 0000000000..e943bf4895
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/config_v2.md
@@ -0,0 +1,346 @@
+---
+id: config_v2
+title: Configuration V2
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Introduction
+
+Over time the YuniKorn configuration has increased in complexity. The current
+configuration for YuniKorn is split over a number of different options, file
+and command line, and formats. There is a requirement now to define an overall
+configuration approach that can be used for all components.
+
+### Goals
+
+- Define a configuration format
+- Define an update and modification strategy
+- Provide consistency between configuration of YuniKorn components
+- Legacy migration strategy
+
+### Non-Goals
+
+- Redefine the queue configuration
+- Migration timeline for the old configuration options
+
+### References
+
+#### Implementation JIRA
+
+[YUNIKORN-1221](https://issues.apache.org/jira/browse/YUNIKORN-1221) - [Umbrella] Configuration V2
+
+#### Other relevant JIRAs
+
+- [YUNIKORN-455](https://issues.apache.org/jira/browse/YUNIKORN-455) - Make the core configurable
+- [YUNIKORN-1213](https://issues.apache.org/jira/browse/YUNIKORN-1213) - Background health checker needs to be configurable
+- [YUNIKORN-1335](https://issues.apache.org/jira/browse/YUNIKORN-1335) - Write yunikorn-web error logs to stderr
+
+### Services to cover
+
+- Scheduler (Standard) - K8Shim + Core
+- Scheduler (Plugin) - K8Shim + Core
+- Admission controller
+
+### Services excluded
+
+ - Web UI (not needed after [YUNIKORN-1335](https://issues.apache.org/jira/browse/YUNIKORN-1335))
+
+## Configuration format
+
+Currently only Kubernetes is a supported deployment pattern. Future shims may
+need an alternative configuration syntax. To enable this, we will update the
+scheduler interface to support passing the core configuration (the content of
+the `queues.yaml` ConfigMap entry) to the core on startup and refresh. The
+existing config watcher functionality in the core will be removed and the shim
+will gain the responsibility for watching for ConfigMap changes. The shim
+actually already has this ability now, it is just not used.
+
+We will use two ConfigMaps to configure YuniKorn:
+ - `yunikorn-defaults`
+ - This will be a new ConfigMap written by Helm; it will contain only values
+ explicitly configured by an administrator via Helm variables.
+ - `yunikorn-configs`
+ - This ConfigMap will contain overrides for yunikorn-defaults. It was
+ previously a Helm hook in previous YuniKorn releases; it will no longer
+ be managed by Helm and may be updated freely by third-party tools.
+
+The effective configuration will be the combination of both `yunikorn-defaults`
+ and `yunikorn-configs`, with `yunikorn-configs` taking precedence.
+
+Neither ConfigMap (even the existing one) is required for proper functionality.
+In the case where a ConfigMap does not exist, we will use compiled-in defaults
+for all options. This enables bootstrapping in a variety of environments,
+including local development mode, as the ConfigMap need not exist to bring the
+scheduler components up. Future configuration options should be set via the
+ConfigMap exclusively.
+
+### Mutability of configuration values
+
+We should strive for hot-reload of all configuration values where possible.
+Where hot-reload is not implemented, this should be clearly documented. For the
+initial release, the core config and admission controller will be
+hot-reloadable but many items in the shim will likely not be. The shim was not
+designed to hot-reload anything so this will be a large future task. Some items
+may never be reloadable due to architectural limitations.
+
+### Update handling
+
+Given the security issues involved with updates, the default Kubernetes access
+control alone should be used for ConfigMap updates. We can still syntax-check
+the core side of the configuration in the admission controller as before.
+Partial updates are allowed as Kubernetes supports patching of ConfigMaps.
+
+### Legacy configuration handling
+
+Helm will be updated to render all existing legacy options into the
+`yunikorn-defaults` ConfigMap for easier transition to the new configuration
+format. Consequently, it is not necessary for YuniKorn components to parse the
+old environment and command-line arguments.
+
+### Environment variables
+
+All environment variables for all services will be removed, with the exception
+of:
+
+ - `NAMESPACE`
+ - Default value: `default`
+ - Will be auto-populated from Helm and resolve to the pod's running
+ namespace
+ - Needed so that YuniKorn can determine what namespace it was launched in
+ - Should only be provided explicitly at development time
+ - `KUBECONFIG`
+ - Default value: `$HOME/.kube/config`
+ - Will be ignored if running inside Kubernetes; in-cluster configuration
+ will be used in that case
+ - Needed so that we can bootstrap to K8s and load the YuniKorn ConfigMap at
+ runtime
+ - Should only be provided explicitly at development time
+
+### Helm considerations
+
+All configuration options which have moved into the ConfigMap will be marked as
+deprecated in the Helm chart documentation. Their replacement is the more
+generic `yunikornDefaults` dictionary. If the legacy values are present but
+the associated `yunikornDefaults` entry is missing, the legacy configuration
+value will be used. This will allow users some time to migrate to the new
+configuration style. The legacy attributes will be removed in a future release.
+
+### Core Configuration
+
+The core scheduler configuration is loaded from the `queues.yaml` section of
+the ConfigMap, assuming `service.policyGroup` is set to the default value
+`queues`. If the policy group has been customized, core scheduler configuration
+will be read from `{service.policyGroup}.yaml`. This section is special as its
+contents are passed directly to the core for evaluation and update and is not
+read directly by the shim. The content of this section is unchanged from prior
+releases.
+
+## Appendix
+
+### Default ConfigMap
+
+If not present, the default ConfigMap will resolve to these defaults.
+Additionally, individual options will resolve to those listed here if not
+specified. Note that these values are subject to change in future releases.
+
+ apiVersion: v1
+ kind: ConfigMap
+ metadata:
+ name: yunikorn-configs
+ data:
+ service.clusterId: "mycluster"
+ service.policyGroup: "queues"
+ service.schedulingInterval: "1s"
+ service.volumeBindTimeout: "10s"
+ service.eventChannelCapacity: "1048576"
+ service.dispatchTimeout: "5m"
+ service.operatorPlugins: "general"
+ service.disableGangScheduling: "false"
+ service.enableConfigHotRefresh: "true"
+ service.placeholderImage: "registry.k8s.io/pause:3.7"
+ service.instanceTypeNodeLabelKey: "node.kubernetes.io/instance-type"
+ health.checkInterval: "30s"
+ log.level: "0"
+ kubernetes.qps: "1000"
+ kubernetes.burst: "1000"
+ admissionController.webHook.amServiceName: "yunikorn-admission-controller-service"
+ admissionController.webHook.schedulerServiceAddress: "yunikorn-service:9080"
+ admissionController.filtering.processNamespaces: ""
+ admissionController.filtering.bypassNamespaces: "^kube-system$"
+ admissionController.filtering.labelNamespaces: ""
+ admissionController.filtering.noLabelNamespaces: ""
+ admissionController.accessControl.bypassAuth: "false"
+ admissionController.accessControl.trustControllers: "true"
+ admissionController.accessControl.systemUsers: "^system:serviceaccount:kube-system:"
+ admissionController.accessControl.externalUsers: ""
+ admissionController.accessControl.externalGroups: ""
+ queues.yaml: |
+ partitions:
+ - name: default
+ placementrules:
+ - name: tag
+ value: namespace
+ create: true
+ queues:
+ - name: root
+ submitacl: '*'`
+
+### Admission Controller Configuration
+
+#### Removed Environment Variables
+ - `POLICY_GROUP`
+ - Use ConfigMap: `service.policyGroup`
+ - `ENABLE_CONFIG_HOT_REFRESH`
+ - Use ConfigMap: `service.enableConfigHotRefresh`
+ - `SCHEDULER_SERVICE_ADDRESS`
+ - Use ConfigMap: `admissionController.webHook.schedulerServiceAddress`
+ - `ADMISSION_CONTROLLER_SERVICE`
+ - Use ConfigMap: `admissionController.webHook.amServiceName`
+ - `ADMISSION_CONTROLLER_NAMESPACE`
+ - Use Environment variable: `NAMESPACE`
+ - `ADMISSION_CONTROLLER_PROCESS_NAMESPACES`
+ - Use ConfigMap: `admissionController.filtering.processNamespaces`
+ - `ADMISSION_CONTROLLER_BYPASS_NAMESPACES`
+ - Use ConfigMap: `admissionController.filtering.bypassNamespaces`
+ - `ADMISSON_CONTROLLER_LABEL_NAMESPACES`
+ - Use ConfigMap: `admissionController.filtering.labelNamespaces`
+ - `ADMISSION_CONTROLLER_NO_LABEL_NAMESPACES`
+ - Use ConfigMap: `admissionController.filtering.noLabelNamespaces`
+
+#### Removed CLI Options
+None
+
+### Scheduler (Standard) Configuration
+
+#### Removed Environment Variables
+ - `USER_LABEL_KEY`
+ - No replacement
+ - `OPERATOR_PLUGINS`
+ - Use ConfigMap: `service.operatorPlugins`
+ - `PLACEHOLDER_IMAGE`
+ - Use ConfigMap: `service.placeholderImage`
+
+#### Removed CLI Options
+ - `-clusterId`
+ - Use ConfigMap: `service.clusterId`
+ - `-clusterVersion`
+ - No replacement
+ - `-policyGroup`
+ - Use ConfigMap: `service.policyGroup`
+ - `-interval`
+ - Use ConfigMap: `service.schedulingInterval`
+ - `-logLevel`
+ - Use ConfigMap: `log.level`
+ - `-logEncoding`
+ - No replacement
+ - `-logFile`
+ - No replacement
+ - `-volumeBindTimeout`
+ - Use ConfigMap: `service.volumeBindTimeout`
+ - `-eventChannelCapacity`
+ - Use ConfigMap: `service.eventChannelCapacity`
+ - `-dispatchTimeout`
+ - Use ConfigMap: `service.dispatchTimeout`
+ - `-kubeQPS`
+ - Use ConfigMap: `kubernetes.qps`
+ - `-kubeBurst`
+ - Use ConfigMap: `kubernetes.burst`
+ - `-operatorPlugins`
+ - Use ConfigMap: `service.operatorPlugins`
+ - `-enableConfigHotRefresh`
+ - Use ConfigMap: `service.enableConfigHotRefresh`
+ - `-disableGangScheduling`
+ - Use ConfigMap: `service.disableGangScheduling`
+ - `-userLabelKey`
+ - No replacement
+ - `-placeHolderImage`
+ Use ConfigMap: `service.placeholderImage`
+
+### Scheduler (Plugin) Configuration
+
+#### Removed Environment Variables
+ - `USER_LABEL_KEY`
+ - No replacement
+ - `OPERATOR_PLUGINS`
+ - Use ConfigMap: `service.operatorPlugins`
+ - `PLACEHOLDER_IMAGE`
+ - Use ConfigMap: `service.placeholderImage`
+
+#### Removed CLI Options
+ - `--yk-scheduler-name`
+ - No replacement
+ - `--yk-cluster-id`
+ - Use ConfigMap: `service.clusterId`
+ - `--yk-cluster-version`
+ - No replacement
+ - `--yk-policy-group`
+ - Use ConfigMap: `service.policyGroup`
+ - `--yk-scheduling-interval
+ - Use ConfigMap: `service.schedulingInterval`
+ - `--yk-kube-config`
+ - Use environment: `KUBECONFIG`
+ - `--yk-log-level`
+ - Use ConfigMap: `log.level`
+ - `--yk-log-encoding`
+ - No replacement
+ - `--yk-log-file`
+ - No replacement
+ - `--yk-event-channel-capacity`
+ - Use ConfigMap: `service.eventChannelCapacity`
+ - `--yk-dispatcher-timeout`
+ - Use ConfigMap: `service.dispatchTimeout`
+ - `--yk-kube-qps`
+ - Use ConfigMap: `kubernetes.qps`
+ - `--yk-kube-burst`
+ - Use ConfigMap: `kubernetes.burst`
+ - `--yk-operator-plugins`
+ - Use ConfigMap: `service.operatorPlugins`
+ - `--yk-enable-config-hot-refresh`
+ - Use ConfigMap: `service.enableConfigHotRefresh`
+ - `--yk-disable-gang-scheduling`
+ - Use ConfigMap: `service.disableGangScheduling`
+ - `--yk-user-label-key`
+ - No replacement
+ - `--yk-placeholder-image`
+ - Use ConfigMap: `service.placeholderImage`
+
+### Helm Configuration
+
+#### Removed Options
+ - `userLabelKey`
+ - No replacement
+
+#### Deprecated Options
+ - `operatorPlugins`
+ - Use ConfigMap: `service.operatorPlugins`
+ - `placeHolderImage`
+ - Use ConfigMap: `service.placeholderImage`
+ - `admissionController.processNamespaces`
+ - Use ConfigMap: `admissionController.filtering.processNamespaces`
+ - `admissionController.bypassNamespaces`
+ - Use ConfigMap: `admissionController.filtering.bypassNamespaces`
+ - `admissionController.labelNamespaces`
+ - Use ConfigMap: `admissionController.filtering.labelNamespaces`
+ - `admissionController.noLabelNamespaces`
+ - Use ConfigMap: `admissionController.filtering.noLabelNamespaces`
+ - `configuration`
+ - Use ConfigMap: `queues.yaml`
+
diff --git a/versioned_docs/version-1.5.1/design/gang_scheduling.md b/versioned_docs/version-1.5.1/design/gang_scheduling.md
new file mode 100644
index 0000000000..44560ef11b
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/gang_scheduling.md
@@ -0,0 +1,605 @@
+---
+id: gang_scheduling
+title: Gang scheduling design
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+# Gang Scheduling Implementation
+A new way of scheduling applications by taking into account the demand for resources the application expects it will generate over time.
+It guarantees the expected demand resources for the application by reserving the resources.
+
+There are two parts to this implementation:
+* Kubernetes Shim
+* Core and scheduling
+
+This document describes the implementation on the core side.
+
+## Document goals
+This document describes the following implementation design points:
+1. Define changes required for the shim to core communication (scheduler interface)
+2. Scheduler storage object changes
+3. Scheduler logic changes
+
+## Excluded design points
+Currently, the Kubernetes shim side implementation is not covered in this design document.
+
+Generalised preemption on the core side will _not_ be discussed in this design.
+
+## Generic flow
+The flow is triggered by a pod that is submitted which triggers the application creation.
+This first pod is in the case of a Spark application, the driver pod.
+In case the flow is triggered from the creation of an application CRD there will not be a first pod.
+This is however outside the core scheduling logic. From the core side there should be no difference between the two cases.
+More details are in the chapter on the [Scheduler logic changes](#scheduler-logic-changes).
+
+The flow of an application submitted. The numbers in the diagram correspond to the description below the diagram.
+
+
+
+Combined flow for the shim and core during startup of an application:
+* An application is submitted with TaskGroup(s) defined. (1)
+* The shim creates the application and passes the application to the core. (2)
+* The shim creates placeholder pods for each of the members of the TaskGroup(s) (3)
+* The pods are processed and passed to the core, as per the normal behaviour, as AllocationAsks for the application with the correct info set. (4)
+* The placeholder AllocationAsk’s are scheduled by the core as if they were normal AllocationAsk’s. (5)
+* All Allocations, even if they are the result of the placeholder AllocationAsks being allocated by the scheduler, are communicated back to the shim.
+* The original real pod is passed to the core as an AllocationAsk. (6)
+* After the real pod and all all the placeholder pods are scheduled the shim starts the real pod that triggered the application creation. (7)
+
+After the first, real, pod is started the following pods should all be handled in the same way (8):
+* A real pod is created on k8s.
+* The pod is processed and an AllocationAsk is created.
+* The scheduler processes the AllocationAsk (more detail below) and replaces a placeholder with the real allocation.
+
+## Application submit handling
+### Total placeholder size
+
+The application if requesting one or more TaskGroups should provide the total size of all the TaskGroup members it is going to request.
+The total resource size is required for the case that the application is scheduled in a queue with a resource limit set.
+
+The value is important for three cases:
+1. gang request is larger than the queue quota
+2. start of scheduling reservations
+3. resource pressure while scheduling reservations
+
+Further detail will be given below in [scheduling in queues with a quota set](#scheduling-in-queues-with-a-quota-set)
+
+The information passed on from the shim should be part of the AddApplicationRequest.
+Detailed information on the build up of the taskGroup(s), or the number of members are not relevant.
+The total resource requested by all taskGroup members is calculated using:
+
+
+
+This total placeholderAsk is added as an optional field to the AddApplicationRequest message.
+The calculation can be made by the shim based on the CRD or annotation provided in the pod description.
+
+If the placeholderAsk is larger than the queue quota set on the queue the application must be rejected.
+This rejection is based on the fact that we cannot in any way honor the request.
+For all other cases the application is accepted and will be scheduled as per normal.
+
+### Handling queue with a FAIR sort policy
+If an application is submitted to a queue that has a FAIR sort policy set it must be rejected.
+Queue sorting for the queue that the application runs in must be set to _FIFO_ or _StateAware_.
+
+Other queue policies cannot guarantee that there is only one _New_ application processed at a time.
+In the case of the _FAIR_ policy we could be allocating multiple _New_ applications at the same time making quota management impossible to enforce.
+The other side effect of using _FAIR_ as a policy could be that we get multiple applications with only a partial allocated guarantee.
+
+Auto-scaling can be triggered due to the fact that the core can not place the placeholders on any node.
+In case the queue would use the _FAIR_ sorting this could lead to other applications taking the scaled up nodes instead of the placeholders again breaking the gang.
+
+## Scheduling in queues with a quota set
+The main case already described above is handling a total placeholder request size that is larger than the quota set on the queue.
+When the application is submitted we can already assess that we cannot satisfy that requirement and reject the request.
+
+In the case that the total placeholder ask does fit in the queue we should not start scheduling until there are enough resources available in the queue to satisfy the total request.
+However this does not stop scheduling of other applications in the queue(s).
+Applications that are already running in the queue could ask for more resources.
+From an application perspective there is no limit set on the resource it can request.
+The gang defined on the application is a guaranteed number of resources, not a maximum number of resources the application can request.
+
+This is complicated by the fact that we have a queue hierarchy.
+There is the possibility that the quota is not set directly on the queue the application is running.
+It could be set on one of the parent queues.
+This case could become complex, and we need to make sure that we keep in mind that we could live lock the scheduling.
+
+In this first phase we should focus on the case that the gang resources requested are also the maximum number of resources the application will request.
+When we look at the queues we should focus on a single queue level with quotas.
+
+These two assumptions are correct for the spark use case without dynamic allocation using a dynamic mapping from a namespace to a queue.
+
+Furthermore, we assume that the quota set on the queue can be totally allocated.
+If the cluster does not have enough resources the cluster will scale up to the size needed to provide all queues with their full quota.
+
+The follow up should add further enhancements for deeper hierarchies and dynamic allocation support.
+This could also leverage preemption in certain use cases, like preempting allocations from applications over their total gang size.
+
+Further enhancements could be added by allowing specifying the time and application will wait for the placeholders to be allocated, or the time to start using the held resources.
+
+## Scheduler logic changes
+The scheduler logic change needs to account for two parts of cycle:
+* The placeholder asks and their allocation.
+* The allocation replacing the placeholder.
+
+The basic assumption is that all pods will generate a placeholder pod request to the core.
+This includes the pod that triggered the application creation if we do not use the application CRD.
+This assumption is needed to make sure that the scheduler core can behave in the same way for both ways of submitting the application.
+The placeholder pods must be communicated to the core before the real pod.
+
+Changes for the placeholder AllocationAsks are the first step.
+As part of the creation of the application the AllocationAsks get added.
+The addition of an AllocationsAsk normally will trigger the application state change as per the scheduling cycle.
+It moves the Application from a _New_ state to an _Accepted_ state. This is as per the current setup, and does not change.
+
+However, in the case that the AllocationAsk has the _placeholder_ flag set the allocation should not trigger a state change, the application stays in _Accepted_ state.
+AllocationAsks are processed until the application has no pending resources.
+AllocationAsks that do not have a _placeholder_ flag set should be ignored as a safety precaution.
+All resulting Allocations for the placeholder pods are confirmed to the shim as per the normal steps.
+This process continues until there are no more placeholder pods to be allocated.
+
+The shim at that point should create the AllocationAsk for the real pod(s) that it has buffered.
+The core cannot and must not assume that there is only one task group per application.
+The core is also not in the position to assume that it has received all AllocationAsks that belong to the task group if option 1 as described above is used by a shim.
+This is also why we have the assumption that every pod creates a placeholder request to the core.
+
+The second change is the replacement of the placeholder pods with the real pods.
+The shim creates an AllocationAsk with the _taskGroupName_ set but the _placeholder_ flag is not set.
+
+The process described here lines up with the process for generic pre-emption.
+An allocation is released by the core and then confirmed by the shim.
+For gang scheduling we have a simple one new to one release relation in the case of pre-emption we can use the same flow with a one new to multiple release relation.
+
+The scheduler processes the AllocationAsk as follows:
+1. Check if the application has an unreleased allocation for a placeholder allocation with the same _taskGroupName._ If no placeholder allocations are found a normal allocation cycle will be used to allocate the request.
+2. A placeholder allocation is selected and marked for release. A request to release the placeholder allocation is communicated to the shim. This must be an async process as the shim release process is dependent on the underlying K8s response which might not be instantaneous.
+ NOTE: no allocations are released in the core at this point in time.
+3. The core “parks” the processing of the real AllocationAsk until the shim has responded with a confirmation that the placeholder allocation has been released.
+ NOTE: locks are released to allow scheduling to continue
+4. After the confirmation of the release is received from the shim the “parked” AllocationAsk processing is finalised.
+5. The AllocationAsk is allocated on the same node as the placeholder used.
+ The removal of the placeholder allocation is finalised in either case. This all needs to happen as one update to the application, queue and node.
+ * On success: a new Allocation is created.
+ * On Failure: try to allocate on a different node, if that fails the AllocationAsk becomes unschedulable triggering scale up.
+6. Communicate the allocation back to the shim (if applicable, based on step 5)
+
+## Application completion
+Application completion has been a long standing issue.
+Currently, applications do not transition to a _completed_ state when done.
+The current states for the application are [documented here](developer_guide/scheduler_object_states.md).
+However, at this point in time an application will not reach the _completed_ state and will be stuck in _waiting_.
+
+This provides a number of issues specifically around memory usage and cleanup of queues in long running deployments.
+
+### Definition
+Since we cannot rely on the application, running as pods on Kubernetes, to show that it has finished we need to define when we consider an application _completed_.
+At this point we are defining that an application is _completed_ when it has been in the _waiting_ state for a defined time period.
+An application enters the waiting state at the time that there are no active allocations (allocated resources > 0) and pending allocation asks (pending resources > 0).
+
+The transition to a _waiting_ state is already implemented.
+The time out of the _waiting_ state is new functionality.
+
+Placeholders are not considered active allocations.
+Placeholder asks are considered pending resource asks.
+These cases will be handled in the [Cleanup](#Cleanup) below.
+
+### Cleanup
+When we look at gang scheduling there is a further issue around unused placeholders, placeholder asks and their cleanup.
+Placeholders could be converted into real allocations at any time there are pending allocation asks or active allocations.
+
+Placeholder asks will all be converted into placeholder allocations before the real allocations are processed.
+
+Entry into the _waiting_ state is already handled.
+If new allocation asks are added to the application it will transition back to a _running_ state.
+At the time we entered the waiting state. there were no pending requests or allocated resources.
+There could be allocated placeholders.
+
+For the entry into the _waiting_ state the application must be clean.
+However, we can not guarantee that all placeholders will be used by the application during the time the application runs.
+Transitioning out of the _waiting_ state into the _completed_ state requires no (placeholder) allocations or asks at all.
+The second case that impact transitions is that not all placeholder asks are allocated, and the application thus never requests any real allocations.
+These two cases could prevent an application from transitioning out of the _accepted_, or the _waiting_ state.
+
+Processing in the core thus needs to consider two cases that will impact the transition out of specific states:
+1. Placeholder asks pending (exit from _accepted_)
+2. Placeholders allocated (exit from _waiting_)
+
+Placeholder asks pending:
+Pending placeholder asks are handled via a timeout.
+An application must only spend a limited time waiting for all placeholders to be allocated.
+This timeout is needed because an application’s partial placeholders allocation may occupy cluster resources without really using them.
+
+An application could be queued for an unknown time, waiting for placeholder allocation to start.
+The timeout for placeholder asks can thus not be linked to the creation of the application or the asks.
+The timeout must start at the time the first placeholder ask is allocated.
+
+The application cannot request real allocations until all placeholder asks are allocated.
+A placeholder ask is also tracked by the shim as it represents a pod.
+Releasing an ask in the core requires a message to flow between the core and shim to release that ask.
+However, in this case the timeout for allocating placeholder asks triggers an application failure.
+When the timeout is triggered and placeholder asks are pending the application will transition from the state it is in, which can only be _accepted_, to _killed_.
+
+The application state for this case can be summarised as:
+* Application status is _accepted_
+* Placeholder allocated resource is larger than zero, and less than the _placeholderAsk_ from the _AddApplicationRequest_
+* Pending resource asks is larger than zero
+
+Entering into the _killed_ state must move the application out of the queue automatically.
+
+The state change and placeholder allocation releases can be handled in a single UpdateResponse message. The message will have the following content:
+* _UpdatedApplication_ for the state change of the application
+* one or more _AllocationRelease_ messages, one for each placeholder, with the _TerminationType_ set to TIMEOUT
+* one or more AllocationAskRelease messages with the _TerminationType_ set to TIMEOUT
+
+The shim processes the AllocationAskRelease messages first, followed by the _AllocationResponse_ messages, and finally the _UpdatedApplication_ message. The application state change to the _killed_ state on the core side is only dependent on the removal of all placeholders pods, not on a response to the _UpdatedApplication_ message.
+
+
+
+Combined flow for the shim and core during timeout of placeholder:
+* The core times out the placeholder allocation. (1)
+* The placeholder Allocations removal is passed to the shim. (2)
+* All placeholder Allocations are released by the shim, and communicated back to the core.
+* The placeholder AllocationAsks removal is passed to the shim. (3)
+* All placeholder AllocationAsks are released by the shim, and communicated back to the core.
+* After the placeholder Allocations and Asks are released the core moves the application to the killed state removing it from the queue (4).
+* The state change is finalised in the core and shim. (5)
+
+Allocated placeholders:
+Leftover placeholders need to be released by the core.
+The shim needs to be informed to remove them. This must be triggered on entry of the _completed_ state.
+After the placeholder release is requested by the core the state transition of the application can proceed.
+The core will process the _AllocationRelease_ messages for placeholder allocations that come back from the shim with the _TerminationType_ set to TIMEOUT as normal without triggering a state change.
+
+The state change and placeholder allocation releases can be handled in a single UpdateResponse message.
+The message will have the following content:
+* _UpdatedApplication_ for the state change of the application
+* zero or more _AllocationRelease_ messages, one for each placeholder, with the _TerminationType_ set to TIMEOUT
+
+The shim processes the _AllocationResponse_ messages first followed by the _UpdatedApplication_ message.
+The application state change to the _completed_ state on the core side is only dependent on the removal of all placeholders pods, not on a response to the _UpdatedApplication_ message.
+
+Entering into the _completed_ state will move the application out of the queue automatically.
+This should also handle the case we discussed earlier around a possible delayed processing of requests from the shim as we can move back from _waiting_ to _running_ if needed.
+A _completed_ application should also not prevent the case that was discussed around cron like submissions using the same application ID for each invocation.
+A _completed_ application with the same application ID must not prevent the submission of a new application with the same ID.
+
+
+
+Combined flow for the shim and core during cleanup of an application:
+* A pod is released at the Kubernetes layer. (1)
+* The shim passes the release of the allocation on to the core. (2)
+* The core transitions the application to a waiting state if no pending or active allocations. (3)
+* The waiting state times out and triggers the cleanup. (4)
+* The placeholder Allocations removal is passed to the shim. (5)
+* All placeholder Allocations are released by the shim, and communicated back to the core.
+* After all placeholders are released the core moves the application to the completed state removing it from the queue (6).
+* The state change is finalised in the core and shim. (7)
+
+## Application recovery
+During application recovery the placeholder pods are recovered as any other pod on a node.
+These pods are communicated to the core by the shim as part of the node as an existing allocation.
+Existing allocations do not have a corresponding _AllocationAsk_ in the core. The core generates an _AllocationAsk_ based on the recovered information.
+
+For gang scheduling the _AllocationAsk_ contains the _taskGroupName_ and _placeholder_ flag.
+During recovery that same information must be part of the _Allocation_ message.
+This is due to the fact that the same message is used in two directions, from the RM to the scheduler and vice versa means we need to update the message and its processing.
+
+If the information is missing from the _Allocation_ message the recovered allocation will not be correctly tagged in the core.
+The recovered allocation will be seen as a regular allocation.
+This means it is skipped as part of the normal allocation cycle that replaces the placeholders.
+
+The logic change only requires that the recovery of existing allocations copies the fields from the interface message into the allocation object in the core.
+
+## Interface changes
+Multiple changes are needed to the communication between the shim and the core to support the gang information needed.
+
+An application must provide the total size of the placeholder requests to prevent accepting an application that can never run.
+
+The current object that is sent from the shim to the core for allocation requests is defined in the AllocationAsk.
+The Allocation, as the result message passed back from the scheduler core does not change. For recovery, which uses the same Allocation message, from the shim to the core, however must contain the gang related fields.
+Gang related fields must be added to both messages.
+
+The allocation release request and response request need to support bidirectional traffic and will need to undergo major changes.
+
+### AddApplication
+The AddApplicationRequest message requires a new field to communicate the total placeholder resource request that will be requested.
+The field is used to reject the application if it is impossible to satisfy the request.
+It can also be used to stop the core from scheduling any real pods for that application until all placeholder pods are processed.
+
+In patched message form that would look like:
+```
+message AddApplicationRequest {
+...
+ // The total amount of resources gang placeholders will request
+ Resource placeholderAsk = 7;
+...
+}
+```
+
+### AllocationAsk
+The first part of the change is the base information for the task group.
+This will require an additional optional attribute to be added.
+The content of this optional attribute is a name, a string, which will be mapped to the name of the task group.
+The field can be present on a real allocation and on a placeholder.
+
+Proposed name for the new field is: _taskGroupName_
+
+To distinguish normal AllocationAsks and placeholder AllocationAsks a flag must be added.
+The flag will never have more than two values and thus maps to a boolean. As the default value for a boolean is _false_ the field should show the fact that it is an AllocationAsk that represents a placeholder as true.
+
+Proposed name for the field is: _placeholder_
+
+In patched message form that would look like:
+```
+message AllocationAsk {
+...
+ // The name of the TaskGroup this ask belongs to
+ string taskGroupName = 10;
+ // Is this a placeholder ask (true) or a real ask (false), defaults to false
+ // ignored if the taskGroupName is not set
+ bool placeholder = 11;
+...
+}
+```
+
+The last part of the task group information that needs to be communicated is the size of the task group.
+This does not require a change in the interface as the current AllocationAsk object can support both possible options.
+
+Requests can be handled in two ways:
+1. Each member of the task group is passed to the core as a separate AllocationAsk with a maxAllocations, or the ask repeat, of 1
+2. The task group is considered one AllocationAsk with a maxAllocations set to the same value as minMember of the task group information.
+
+With option A the shim will need to generate multiple AllocationAsk objects and pass each to the core for scheduling.
+Each AllocationAsk is linked to one pod.
+Option B will only generate one AllocationAsk for all placeholder pods.
+Option B requires less code and has less overhead on the core side.
+However the logic on the shim side might be more complex as the returned allocation needs to be linked to just one pod.
+
+Proposal is to use option: A
+
+### Allocation
+Similar to the change for the _AllocationAsk_ the _Allocation_ requires additional optional attributes to be added.
+The new fields distinguish a normal Allocation and placeholder Allocations on recovery.
+The same rules apply to these fields as the ones added to the _AllocationAsk_.
+
+The content of this optional attribute is a name, a string, which will be mapped to the name of the task group.
+The field can be present on a real allocation and on a placeholder.
+
+Proposed name for the new field is: _taskGroupName_
+
+The flag will never have more than two values and thus maps to a boolean.
+As the default value for a boolean is _false_ the field should show the fact that it is an Allocation that represents a placeholder as true.
+
+Proposed name for the field is: _placeholder_
+
+In patched message form that would look like:
+```
+message Allocation {
+...
+ // The name of the TaskGroup this allocation belongs to
+ string taskGroupName = 11;
+ // Is this a placeholder allocation (true) or a real allocation (false), defaults to false
+ // ignored if the taskGroupName is not set
+ bool placeholder = 12;
+...
+}
+```
+
+### AllocationRelease Response and Request
+The name for the messages are based on the fact that the release is always triggered by the shim.
+In case of preemption and or gang scheduling the release is not triggered from the shim but from the core.
+That means the message name does not cover the usage. A response message might not have an associated request message.
+It could be used to indicate direction but that is in this case confusing.
+
+When a release is triggered from the core, for preemption or the placeholder allocation, a response is expected from the shim to confirm that the release has been processed.
+This response must be distinguished from a request to release the allocation initiated by the shim.
+A release initiated by the shim must be followed by a confirmation from the core to the shim that the message is processed.
+For releases initiated by the core no such confirmation message can or must be sent.
+In the current request message there is no way to indicate that it is a confirmation message.
+
+To fix the possible confusing naming the proposal is to merge the two messages into one message: _AllocationRelease_.
+
+The _AllocationReleaseRequest_ is indirectly part of the _UpdateRequest_ message as it is contained in the _AllocationReleasesRequest_.
+The _AllocationReleaseResponse_ is part of the _UpdateResponse_ message.
+The flow-on effect of the rename and merge of the two messages is a change in the two messages that contain them.
+The message changes for _UpdateResponse_ and _AllocationReleasesRequest_ are limited to type changes of the existing fields.
+
+| Message | Field ID | Old type | New type |
+| ------------------------- | -------- | ------------------------- | ----------------- |
+| UpdateResponse | 3 | AllocationReleaseResponse | AllocationRelease |
+| AllocationReleasesRequest | 1 | AllocationReleaseRequest | AllocationRelease |
+
+In patched message form that would look like:
+```
+message UpdateResponse {
+...
+ // Released allocation(s), allocations can be released by either the RM or scheduler.
+ // The TerminationType defines which side needs to act and process the message.
+ repeated AllocationRelease releasedAllocations = 3;
+...
+}
+
+message AllocationReleasesRequest {
+ // Released allocation(s), allocations can be released by either the RM or scheduler.
+ // The TerminationType defines which side needs to act and process the message.
+ repeated AllocationRelease releasedAllocations = 1;
+...
+}
+```
+
+The merged message _AllocationRelease_ will consist of:
+
+| Field name | Content type | Required |
+| --------------- | ----------------- | -------- |
+| partitionName | string | yes |
+| applicationID | string | no |
+| UUID | string | no |
+| terminationType | _TerminationType_ | yes |
+| message | string | no |
+
+Confirmation behaviour of the action should be triggered on the type of termination received.
+The core will confirm the release to the shim of all types that originate in the shim and vice versa.
+
+A confirmation or response uses the same _TerminationType_ as was set in the original message.
+An example of this is a pod that is removed from K8s will trigger an _AllocationRelease_ message to be sent from the shim to the core with the TerminationType STOPPED_BY_RM. The core processes the request removing the allocation from the internal structures, and when all processing is done it responds to the shim with a message using the same _TerminationType_.
+The shim can ignore that or make follow up changes if needed.
+
+A similar process happens for a release that originates in the core.
+Example of the core sending an _AllocationRelease_ message to the shim using the _TerminationType_ PREEMPTED_BY_SCHEDULER.
+The shim handles that by releasing the pod identified and responds to the core that it has released the pod.
+On receiving the confirmation that the pod has been released the core can progress with the allocation and preemption processing.
+
+In patched message form that would look like:
+```
+message AllocationRelease {
+ enum TerminationType {
+ STOPPED_BY_RM = 0;
+ TIMEOUT = 1;
+ PREEMPTED_BY_SCHEDULER = 2;
+ PLACEHOLDER_REPLACED = 3;
+ }
+
+ // The name of the partition the allocation belongs to
+ string partitionName = 1;
+ // The application the allocation belongs to
+ string applicationID = 2;
+ // The UUID of the allocation to release, if not set all allocations are released for
+ // the applicationID
+ string UUID = 3;
+ // The termination type as described above
+ TerminationType terminationType = 4;
+ // human-readable message
+ string message = 5;
+}
+```
+### TerminationType
+The currently defined _TerminationType_ values and specification of the side that generates (Sender), and the side that actions and confirms processing (Receiver):
+
+| Value | Sender | Receiver |
+| ------------------------ | ------ | -------- |
+| STOPPED_BY_RM | shim | core |
+| TIMEOUT * | core | shim |
+| PREEMPTED_BY_SCHEDULER * | core | shim |
+
+* currently not handled by the shim, core or both
+
+When the placeholder allocation gets released the _AllocationReleaseResponse_ is used to communicate the release back from the core to the shim.
+The response contains an enumeration called _TerminationType_, and a human-readable message.
+For tracking and tracing purposes we should add a new _TerminationType_ specifically for the placeholder replacement. The shim must take action based on the type and confirm the allocation release to the core.
+
+It should provide enough detail so we do not have to re-use an already existing type, or the human-readable message.
+The human-readable format can still be used to provide further detail on which new allocation replaced the placeholder.
+
+Proposal is to add: _PLACEHOLDER_REPLACED_
+
+| Value | Sender | Receiver |
+| ---------------------| ------ | -------- |
+| PLACEHOLDER_REPLACED | shim | core |
+
+As part of the Scheduler Interface cleanup ([YUNIKORN-486](https://issues.apache.org/jira/browse/YUNIKORN-486)) the _TerminationType_ should be extracted from the _AllocationRelease_ and _AllocationaskRelease_ message.
+It is an enumeration that can be shared between multiple objects.
+[YUNIKORN-547](https://issues.apache.org/jira/browse/YUNIKORN-547) has been logged to handle this as it has an impact on the code outside of the scope of gang scheduling.
+
+
+### AllocationAskRelease Response and Request
+The allocation ask release right now can only be triggered by the shim.
+In order for the core to perform the cleanup when the placeholder allocation times out, we need to make this a bidirectional message.
+Similarly to the Allocation we would rename the _AllocationAskReleaseRequest_ to _AllocationAskRelease_, so we can use this message in both directions:
+```
+message AllocationReleasesRequest {
+...
+ // Released allocationask(s), allocationasks can be released by either the RM or
+ // scheduler. The TerminationType defines which side needs to act and process the
+ // message.
+ repeated AllocationAskRelease allocationAsksToRelease = 2;
+}
+```
+
+Similar processing logic based on the _TerminationType_ which is used for allocations should be used for ask releases.
+In patched message form that would look like:
+```
+message AllocationAskRelease {
+ enum TerminationType {
+ STOPPED_BY_RM = 0;
+ TIMEOUT = 1;
+ PREEMPTED_BY_SCHEDULER = 2;
+ PLACEHOLDER_REPLACED = 3;
+ }
+...
+ // The termination type as described above
+ TerminationType terminationType = 4;
+...
+}
+```
+
+Confirmation behaviour of the action should be triggered on the type of termination received.
+The core will confirm the release to the shim of all types that originate in the shim and vice versa.
+
+A confirmation or response uses the same _TerminationType_ as was set in the original message.
+
+## Scheduler storage object changes
+### AllocationAsk
+In line with the changes for the communication the objects in the scheduler also need to be modified to persist some of the detail communicated.
+The AllocationAsk that is used in the communication has an equivalent object inside the scheduler with the same name.
+This object needs to be able to store the new fields proposed above.
+
+Proposed new fields: _taskGroupName_ and _placeholder_.
+
+In the current interface specification a field called _executionTimeoutMilliSeconds_ is defined.
+This is currently not mapped to the object inside the scheduler and should be added.
+Time or Duration are stored as native go objects and do not include a size specifier.
+
+Proposed new field: _execTimeout_
+
+### Allocation
+After the allocation is made an Allocation object is created in the core to track the real allocation. This Allocation object is directly linked to the application and should show that the allocation is a placeholder and for which task group. This detail is needed to also enable the correct display of the resources used in the web UI.
+
+The propagation of the placeholder information could be achieved indirectly as the allocation object references an AllocationAsk. This would require a lookup of the AllocationAsk to assess the type of allocation. We could also opt to propagate the data into the Allocation object itself. This would remove the lookup and allow us to directly filter allocations based on the type and or task group information.
+
+From a scheduling and scheduler logic perspective the indirect reference is not really desirable due to the overhead of the lookups required. This means that the same fields added in the AllocationAsk are also added to the Allocation object.
+
+Proposed new fields: _taskGroupName_ and _placeholder_.
+
+To support the release of the allocation being triggered from the core tracking of the release action is required. The release is not final until the shim has confirmed that release. However during that time period the allocation may not be released again.
+
+Proposed new field: _released_
+
+At the point that we replace the placeholder with a real allocation we need to release an existing placeholder.
+The Allocation object allows us to specify a list of Allocations to release.
+This field was added earlier to support preemption.
+This same field will be reused for the placeholder release.
+
+### Application
+The AddApplicationRequest has a new field added that needs to be persisted in the object inside the scheduler.
+
+Proposed new field: _placeholderAsk_
+
+In the current interface specification a field called _executionTimeoutMilliSeconds_ is defined. This is currently not mapped to the object inside the scheduler and should be added. Time or Duration are stored as native go objects and do not include a size specifier.
+
+Proposed new field: _execTimeout_
+
+The application object should be able to track the placeholder allocations separately from the real allocations. The split of the allocation types on the application will allow us to show the proper state in the web UI.
+
+Proposed new field: _allocatedPlaceholder_
+
+
+### Queue & Node
+No changes at this point.
+The placeholder allocations should be counted as “real” allocations on the Queue and Node.
+By counting the placeholder as normal the quota for the queue is enforced as expected.
+The Node object needs to also show normal usage to prevent interactions with the autoscaler.
diff --git a/versioned_docs/version-1.5.1/design/generic_resource.md b/versioned_docs/version-1.5.1/design/generic_resource.md
new file mode 100644
index 0000000000..f4a8ce448c
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/generic_resource.md
@@ -0,0 +1,69 @@
+---
+id: generic_resource
+title: Generic Resource Types in Namespace Quota
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+# Generic Resource Types in Namespace Quota
+Tracking jira: [YUNIKORN-1275](https://issues.apache.org/jira/browse/YUNIKORN-1279)
+
+## Functional gap
+The queue configuration allows all resource types to be set in a quota. Namespace annotations do not. Support for the same resource types in the annotations on namespaces should be possible.
+
+## Current solution
+In the current setup YuniKorn supports annotations on a namespace to specify a resource quota for that namespace. This is used in combination with placement rules to create a quota limited queue automatically based on the namespace in Kubernetes.
+
+The annotations that are supported limit the possible resource types that are supported on these auto created queues. Each resource type uses its own annotation. Current annotations supported as per the quota management documentation:
+```
+yunikorn.apache.org/namespace.max.cpu
+yunikorn.apache.org/namespace.max.memory
+```
+The queue configuration itself, as part of the yaml file, supports all Kubernetes resources including extended resources.
+## Proposed solution
+The current solution uses a specific annotation for each type that is supported. This means that each new resource would require a new annotation to be defined. Reading a new annotation requires a code change in the k8shim.
+
+In comparison when we look at the gang scheduling setup with the task groups specification we are far more flexible. In that case we allow a map of resources to be specified. The map uses the resource name as the key and allows a value as per the Kubenetes resource specification. This solution allows any resource type to be set as a request for a task group.
+
+An equivalent solution should be allowed for the quota annotation on the namespace. This would provide a more flexible solution that does not require code changes for every new resource type that must be supported as part of the namespace quota.
+
+### Annotation name
+The new name for the annotation should not interfere with the existing annotations that are used for the memory and cpu resource quota. Beside that rule we are free to use any name that complies with the naming conventions for names.
+
+The proposal is to use:
+```
+yunikorn.apache.org/namespace.quota
+```
+### Annotation content
+The content of the annotation must be a simple string. There are no length limits for a specific annotation. All annotations together on one object do have a size limit however that is not a restriction we have to plan around.
+
+Since the content must be a simple string we should use a simple json representation for the quota that contains a list of resources. Representing the quota:
+```yaml
+yunikorn.apache.org/namespace.quota: "{\"cpu\": \"100m\",\"memory\": \"1G\",\"nvidia.com/gpu\": \"1\"}"
+```
+
+Similar as for other resources we allow in annotations: we allow any string as the key content.
+The value content should be interpreted as a Kubernetes formatted resource quantity. Parsing will handle that enforcement. If any of the values do not comply with the formatting no quota will be set.
+Propagation to the core
+No changes are proposed or required. The quota is passed from the k8shim into the core via the application tags. The content of the tag is a Resource object as defined in the scheduler interface. The schedule interface Resource object supports arbitrary resources already. The content passed from the k8shim to the core will not change. There will also be no changes in the way the quota will be processed in the core as that processing is not linked to resource types.
+Backwards compatibility
+The current annotations will remain supported for the 1.x minor releases. Deprecation will be announced with the first release that supports the new annotation. Messages mentioning the processing of the old annotation will also be logged at a WARN level in the logs.
+
+Removing the existing annotation processing is a breaking change that could cause a large change in behaviour. Removal of processing for the old annotations should be part of the next major release. The next major release is 2.0.0. This is based on the fact that we do not have a deprecation policy defined as yet.
+
+Preference in processing will be with the new annotations. In the case that both the old and new annotations are present on the namespace the new annotation will be used. Using both old and new annotations, i.e. merging of the two sets, will not be supported.
diff --git a/versioned_docs/version-1.5.1/design/historical_usage_tracking.md b/versioned_docs/version-1.5.1/design/historical_usage_tracking.md
new file mode 100644
index 0000000000..3e0b447f27
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/historical_usage_tracking.md
@@ -0,0 +1,759 @@
+---
+id: historical_usage_tracking
+title: Historical Usage Tracking
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+
+## Introduction
+
+Tracking an application lifecycle is not easy in the current YuniKorn setup.
+Logs and statedump data is required to build an overview.
+Historical tracking of applications is only possible with limited detail per application.
+Applications come and go, we only track what is running and a time limited set of completed applications.
+
+Storing detailed tracking information in memory is problematic.
+Unbound growth can cause performance issues or even an out of memory failure of the service.
+The lessons learned from the YARN days have shown that it is not simple.
+
+A conscious choice was also made to keep YuniKorn stateless.
+The service does not maintain its own datastore for any of its operations.
+Adding a datastore for application and usage tracking will complicate things.
+
+Tracking data can be easily generated and made available for external consumption.
+The design will discuss the option to generate an event stream for applications, queues and nodes to allow external tools to build an overview of the usage pattern.
+
+Work to be tracked under [YUNIKORN-1628](https://issues.apache.org/jira/browse/YUNIKORN-1628)
+
+## Goals
+
+- Implementation of an event stream for an application, including:
+ - State changes
+ - Asks and allocation changes
+- Implementation of an event stream for a node, including:
+ - State changes
+ - Allocation changes
+- Implementation of an event stream for a queue, including:
+ - State changes
+ - Usage changes
+- Define a REST interface for event retrieval
+
+## Non Goals
+
+- Add a data store for the historical data
+- Display the event information
+- Rebuild data on recovery
+ - Historical data will not be rebuild
+- Authentication and Authorisation on the REST interface
+
+## Existing Event System
+
+The event system was designed to be far more flexible than the current usage.
+Events for requests, applications, nodes and queues have been defined.
+Most of those are currently not used.
+The current event system is built on top of a generic event definition.
+The scheduler interface defines the event as:
+
+```go
+si.EventRecord {
+ Type: recordType,
+ ObjectID: String,
+ GroupID: String,
+ Reason: String,
+ Message: String,
+ TimestampNano: int64,
+}
+```
+
+Events definitions are used in the core and shim.
+The simplicity of the events will most likely not match the current requirements.
+
+As part of the standard scheduling cycle we track details at different levels.
+Prometheus' metrics are tracked for specific changes and updates.
+We also generate events, REQUEST events, that get passed back into the K8shim to attach to the Kubernetes pods.
+
+The current events that get passed back are for:
+
+- *Insufficient Queue Resources*: part of the allocation cycle. Called from: `Application.tryAllocate()`
+- *Placeholder Size Mismatch*: part of the placeholder replacement cycle. Called from: `Application.tryPlaceholderAllocate()`
+
+Both events are of the type REQUEST.
+The K8shim also implements processing of NODE events but there is no code that generates those types of events.
+The APPLICATION and QUEUE type events are not created or processed.
+
+The event system was added as part of [YUNIKORN-42](https://issues.apache.org/jira/browse/YUNIKORN-42).
+The jira contains a simple design document for the event system.
+It only was implemented for events that we could not at that point get in any other way: [design v2](https://docs.google.com/document/d/1aKfY6wnBPCyBl03UfmMHbTSpabAbfxtsHT2KzOgEOQs/edit).
+The original thought was a more generic event system.
+It would relate everything back to either a pod or the YuniKorn CRD and was focussed on diagnostics and troubleshooting in general: [design v1](https://docs.google.com/document/d/19iMkLJGVwTSq9OfV9p75wOobJXAxR_CRo4YzSRG_Pzw/edit#heading=h.worp3vfnqgtn).
+Linking it all back to a specific pod is The YuniKorn CRD is not used at this point in time.
+
+### Scheduler integration
+
+When the event system was designed the whole purpose was to allow out of band processing of the events from the scheduling cycle.
+The events are generated during the scheduling cycle and processed asynchronously in the event system.
+
+A channel is used for collecting the events during the scheduling cycle.
+The scheduler generates an event and adds it for processing to the channel.
+After the placement of the event on the channel the scheduler proceeds with the normal cycle.
+Processing of the events does not, and must not, block the scheduling cycle.
+
+This part of the event system must be maintained as it will guarantee the performance of the scheduler.
+
+### Event storage
+
+All events that have been sent to the channel are read from the channel and placed in a temporary store for publishing.
+The store is a simple map with the key of the *ObjectID*.
+
+Some of the assumptions in the event store however make it not directly usable for the purpose as described here.
+The main limitation is that there can only be one event in the channel per *ObjectID*.
+The newest one is kept, the older one is dropped when a new event comes in.
+This however does not mean the code already available with minor changes could be re-used for this purpose.
+
+### Publishing events
+
+The current event system uses a push system for event publishing.
+The event system allows the creation of multiple event publishers.
+There is no implementation or design for a pull system, like for instance a REST based interface.
+
+Currently, there is only an event publisher for a shim defined.
+This push system will send all the events that have been collected during the push interval to the shim.
+All events that are pushed will be removed from the store.
+This keeps the size of the store to a minimum.
+
+There is no filtering or post-processing of events implemented.
+Each event stored in the store is forwarded to the shim when the publisher runs.
+See limitations of the store mentioned above.
+In the earlier design a level filter was described.
+That level was never part of the follow-up design, not in the events and not in the processing.
+
+## Event System Updates
+
+The currently defined events are not a fit for the system we want and need.
+Since the events are only used between the core and the K8shim via the publisher we do not need to maintain backwards compatibility.
+Changes can be made to the messages as we do not currently expose the message to the outside.
+
+The *message* and *reason* fields are currently not properly used.
+The content of the two overlaps.
+The message in both cases contains the reason, in slightly different wording.
+We do not need both.
+
+### Event description
+
+We are also missing two fields to allow an extended usage for historical tracking: the *resource* and the *change type*.
+Based on that the new message that would allow using the event system for tracking historical change would be:
+
+```go
+si.EventRecord {
+ Type: eventRecordType,
+ ChangeType: eventChangeType,
+ ChangeDetail: eventChangeDetail,
+ TimestampNano: int64,
+ ObjectID: String,
+ ReferenceID: String,
+ Resource: si.Resource,
+ Message: String,
+}
+```
+
+The existing *GroupID* has been renamed to *ReferenceID*. The
+*ReferenceID* is the identification of the second object for the event.
+As an example that would be the Allocation UUID for a new allocation
+added to an application, request or node. For the queue that would be
+the application ID.
+
+Note that the order of the attributes in the final definition might be
+different as we do not reuse names and IDs in the messages.
+
+### Event types
+
+By keeping the event type for REQUESTS we can still fulfil the original
+design of YUNIKORN-42. The current enumeration for the *eventRecordType*
+would not change. Definition of the eventRecordType enumeration:
+
+```text
+UNKNOWN = 0
+REQUEST = 1
+APP = 2
+NODE = 3
+QUEUE = 4
+```
+
+### Change types
+
+The additional change type that is added allows us to track the type of
+change. Depending on the content of the other fields it provides the
+possibility to track all changes we need to track. Definition of the
+eventChangeType enumeration:
+
+```text
+NONE = 0
+SET = 1
+ADD = 2
+REMOVE = 3
+```
+
+### Change details
+
+Change detail provides more on the reason for the event. The change
+detail is an enumerated value linked to the event types.
+
+```text
+DETAILS_NONE = 0
+
+REQUEST_CANCEL = 100 // Request cancelled by the RM
+REQUEST_ALLOC = 101 // Request allocated
+REQUEST_TIMEOUT = 102 // Request cancelled due to timeout
+
+APP_ALLOC = 200 // Allocation changed
+APP_REQUEST = 201 // Request changed
+APP_REJECT = 202 // Application rejected on create
+APP_NEW = 203 // Application added with state new
+APP_ACCEPTED = 204 // State change to accepted
+APP_RUNNING = 206 // State change to running
+APP_COMPLETING = 207 // State change to completing
+APP_COMPLETED = 208 // State change to completed
+APP_FAILING = 209 // State change to failing
+APP_FAILED = 210 // State change to failed
+APP_RESUMING = 211; // State change to resuming
+APP_EXPIRED = 212; // State change to expired
+
+NODE_DECOMISSION = 300 // Node removal
+NODE_READY = 301 // Node ready state change
+NODE_SCHEDULABLE = 302 // Node schedulable state change (cordon)
+NODE_ALLOC = 303 // Allocation changed
+NODE_CAPACITY = 304 // Capacity changed
+NODE_OCCUPIED = 305 // Occupied resource changed
+NODE_RESERVATION = 306; // Reservation/unreservation occurred
+
+QUEUE_CONFIG = 400 // Managed queue update or removal
+QUEUE_DYNAMIC = 401 // Dynamic queue update or removal
+QUEUE_TYPE = 402 // Queue type change
+QUEUE_MAX = 403 // Max resource changed
+QUEUE_GUARANTEED = 404 // Guaranteed resource changed
+QUEUE_APP = 405 // Application changed
+QUEUE_ALLOC = 406 // Allocation changed
+ALLOC_CANCEL = 500 // Allocation cancelled by the RM
+ALLOC_PREEMPT = 501 // Allocation preempted by the core
+ALLOC_TIMEOUT = 502 // Allocation cancelled due to timeout
+ALLOC_REPLACED = 503 // Allocation replacement (placeholder)
+ALLOC_NODEREMOVED = 504 // Allocation cancelled, node removal
+```
+
+### Content definition
+
+The *eventRecordType* defines the object that the *ObjectID* points to.
+The content of the *ReferenceID* depends on two things:
+- Change type
+- Object type inferred by the *eventRecordType*
+
+For an object of type application it would not make sense to have the reference point to another application etc.
+At this point the following mapping for *ReferenceID* is assumed:
+
+| Event Type | Reference ID |
+|------------|----------------------------------|
+| REQUEST | ApplicationID |
+| APP | Allocation UUID or Request UUID |
+| NODE | Allocation UUID |
+| QUEUE | ApplicationID or Allocation UUID |
+
+If the *eventChangeType* Is NONE or SET the *ReferenceID* is always empty.
+The exception is for the REQUEST as in that case the change type NONE has an application ID set.
+This special case is used to implement the existing functionality of the event system: sending events to the shim.
+
+The APP record type supports Request IDs and Allocation UUIDs to be set in the *ReferenceID*.
+The type of ID that is referenced is defined by the *ChangeDetail*.
+
+The QUEUE record type supports application IDs and Allocation UUIDs to be set in the *ReferenceID*.
+Individual allocations are not tracked on the queue.
+However, we leave that option open for the event system at this point.
+
+For the QUEUE if *ReferenceID* and *Resource* are set, the ID points to an allocation.
+This can only happen if the *ChangeDetail* is set to QUEUE_ALLOC.
+
+If only the *ReferenceID* is set it points to an application.
+This can only happen if the *ChangeDetail* is set to QUEUE_APP.
+Both cases can use the same *eventChangeType*, add or remove an allocation or an application from a queue.
+
+The *Resource* that is part of any event would be the size of the resource for that change.
+The interpretation depends on the *eventChangeType*.
+For a SET it is the absolute value. For the ADD and REMOVE it is the positive delta on the existing tracked value.
+The *Resource* is always a positive value.
+
+## Event storage
+
+The storage of events needs to be updated to allow multiple events to be stored for an *ObjectID*.
+This is required as one object could trigger multiple events in a short time and all are important to track.
+This will require a change from a map based storage to a different store or a change in the key used for the map.
+
+The simplest solution might be a slice of pointers to events.
+The slice has a configurable, limited set of entries.
+The slice is accessed and maintained as a ring buffer.
+This will prevent an unlimited growth of the schedulers memory requirements.
+During the implementation a decision will be made which configuration, time; count or both, for limiting the size will be supported.
+
+Nil, or empty, events will not be added to the storage.
+However, no publisher must assume that while processing events retrieved from the buffer and needs to handle nil references.
+A location in the buffer could contain a nil pointer and must be skipped while iterating over the buffer.
+If a publisher gets a nil event pointer it must not crash and proceed as normal.
+
+The current design will use two stores: one for external retrieval and one for the shim events.
+A filter at the receiving end will be created.
+The shim events will then be processed as per the existing implementation.
+The new events will be stored and managed as per this design document in [SHIM events](#shim-events).
+
+### Publishing events
+
+Events are currently not post processed and all events are sent to the shim.
+With the newly added events the publishing needs to be split into two separate loops.
+
+#### SHIM events
+
+Current functionality will be supported as described above.
+Updates are required as the content of the events has changed.
+The same event type will be generated.
+The same processing mechanism will be used.
+
+As part of processing events REQUEST type events are special.
+The REQUEST type with the change type NONE has an application ID set and is only sent as an event to the shim.
+These events must not be made available by any other publishers and are stored in a separate ring buffer.
+After the event is sent to the shim the reference to the event in the buffer is cleared, replaced with a nil.
+
+#### REST
+
+The REST service that is currently running can be reused for exposing the events.
+Proposal is to add a new endpoint to the HTTP service.
+The REST endpoint servicing the event data should be a new end point.
+
+Based on the current REST api definition the data exposed in the following new endpoint will be added to expose the events:
+```text
+/ws/v1/events/batch
+```
+
+The design will not define a sophisticated interface for querying events, like filtering, grouping, etc.
+This is a deliberate decision.
+Such an API can encourage bad practices later on.
+That should be done in a separate application which fetches the events from YuniKorn and persists them in a permanent storage and offers a more feature rich REST/query interface.
+This would be similar to what already exists in Apache Hadoop like Job History Server or Application Timeline Service.
+
+As mentioned earlier in this document, such an application is not in the scope.
+
+The batch endpoint, by default, returns a limited number of events.
+The number of events to return can be specifically set via the query parameter *count*.
+If the requested *count* is larger than the available number of events all events are returned.
+
+```text
+/ws/v1/events/batch?count=100
+```
+
+To prevent a DOS attack the maximum value of *count* might be limited as part of the implementation.
+
+The second query parameter that will be supported is *start*.
+This specifies the start ID of the first event returned.
+Every event is assigned a unique id starting from 0.
+The ring buffer maintains the current available highest and lowest ID.
+If *start* refers to an ID which doesn't exist in the buffer, then an empty response is returned from the server, with *LowestID* and *HighestID* properly filled:
+```json lines
+{
+ "InstanceUUID": "b4751b7d-a1a3-4431-bb68-bff896adb9c2",
+ "LowestID": 1100,
+ "HighestID": 11000,
+ "EventRecords": null
+}
+```
+
+In the next request, start should be defined as 1100.
+
+In the response, the *InstanceUUID* field shows the unique instance ID.
+Since YuniKorn is stateless, the generated ID for an event is not saved anywhere.
+If an event consumer saves the events to a backend database, it makes it possible to distinguish between events with the same ID.
+Also, clients can easily detect that YuniKorn was restarted.
+
+#### Streaming
+
+Streaming allows the user to see a continuous stream of events as they are occurring.
+This can be very useful if we want to trace the state changes inside YuniKorn for a longer amount of time.
+The REST interface can return the last "n" amount of events that occurred in a certain timeframe, but it will always be limited.
+We can only increase the number of events at the expense of memory usage, which might not be acceptable in certain environments.
+
+Although streaming is a nice feature, we do not consider it as a "must have", at least not in the first version of the history tracking.
+This is because it's more complicated in nature: it maintains an open HTTP connection towards the client which makes it stateful.
+There are ordering and memory usage concerns&considerations.
+We need to make sure that rogue clients cannot abuse it.
+Therefore, the first release of history tracking focuses on the batch interface.
+
+Streaming needs to coexist beside the current REST api.
+A separate endpoint must be exposed for the event stream:
+```text
+/ws/v1/events/stream
+```
+
+Streaming can be similar to how it's implemented inside Kubernetes informers.
+The API server keeps the connection alive and sends the necessary events about pods, nodes, configmaps, etc.
+The incoming data stream is decoded by the listeners.
+Event processing on the client side is not part of this design.
+
+At this point we do not provide endpoints for consumers to stream a specific event type as defined in [Event types](#event-types).
+This could be made available via separate endpoints in the future following the same design as for *batch* specification
+
+The *stream* endpoint does not take any query parameters.
+The consumer must allow for more than one (1) event to be sent in one response.
+No information besides the events will be sent in the response.
+
+As an example below the approximate output for the stream endpoint for two events
+
+```json
+[
+ {
+ "type": 2,
+ "changeType": 1,
+ "changeDetail": 203,
+ "timestamp": 1649167576110750000,
+ "objectID": "spark-app-1"
+ },
+ {
+ "type": 4,
+ "changeType": 2,
+ "changeDetail": 405,
+ "timestamp": 1649167576110754000,
+ "objectID": "root.spark",
+ "referenceID": "spark-app-1"
+ }
+]
+```
+
+## Event overview
+
+### Generic events
+
+The following events are generated for streaming and REST consumers, meaning that they will not be sent to the shim.
+This is based on the description of the [Event System Updates](#event-system-updates) earlier in this document.
+It serves as a reference for the core scheduler actions that will trigger the event.
+
+| Event type | Change type | Change details | Reference type | Notes |
+|------------|-------------|-------------------|----------------|--------------------------------------------------|
+| APP | ADD | DETAILS_NONE | | New application added |
+| APP | ADD | APP_ALLOC | AllocationID | Successful allocation |
+| APP | ADD | APP_REQUEST | RequestID | Incoming resource request (pod) |
+| APP | REMOVE | DETAILS_NONE | | Normal removal of application |
+| APP | REMOVE | APP_REJECT | | Application rejected |
+| APP | REMOVE | ALLOC_CANCEL | AllocationID | Normal removal requested by the shim |
+| APP | REMOVE | ALLOC_TIMEOUT | AllocationID | Timeout |
+| APP | REMOVE | ALLOC_REPLACED | AllocationID | Replacement (placeholder) |
+| APP | REMOVE | ALLOC_PREEMPT | AllocationID | Preemption triggered |
+| APP | REMOVE | ALLOC_NODEREMOVED | AllocationID | Removal triggered by node removal |
+| APP | REMOVE | APP_REQUEST | RequestID | Normal removal requested by the shim |
+| APP | REMOVE | REQUEST_TIMEOUT | RequestID | Timeout (placeholder) |
+| APP | REMOVE | REQUEST_CANCEL | RequestID | Removal triggered by application removal |
+| APP | SET | APP_NEW | | State change: New |
+| APP | SET | APP_ACCEPTED | | State change: Accepted |
+| APP | SET | APP_RUNNING | | State change: Running |
+| APP | SET | APP_COMPLETING | | State change: Completing |
+| APP | SET | APP_COMPLETED | | State change: Completed |
+| APP | SET | APP_FAILING | | State change: Failing |
+| APP | SET | APP_FAILED | | State change: Failed |
+| APP | SET | APP_RESUMING | | State change: Resuming |
+| APP | SET | APP_EXPIRED | | State change: Expired |
+| | | | | |
+| NODE | ADD | DETAILS_NONE | | New node added to the cluster |
+| NODE | ADD | NODE_ALLOC | AllocationID | Successful allocation |
+| NODE | REMOVE | NODE_DECOMISSION | | Removal requested by the shim |
+| NODE | REMOVE | NODE_ALLOC | AllocationID | Normal allocation removal requested by the shim |
+| NODE | SET | NODE_READY | | Update "ready" status |
+| NODE | SET | NODE_SCHEDULABLE | | Update "schedulable" status |
+| NODE | SET | NODE_CAPACITY | | Update node capacity |
+| NODE | SET | NODE_OCCUPIED | | Update occupied resources |
+| NODE | ADD | NODE_RESERVATION | | Add reservation to a node |
+| NODE | REMOVE | NODE_RESERVATION | | Remove reservation from a node |
+| | | | | |
+| QUEUE | ADD | DETAILS_NONE | | Adding new configured queue |
+| QUEUE | ADD | QUEUE_DYNAMIC | | Adding new dynamic queue |
+| QUEUE | ADD | QUEUE_APP | ApplicationID | Application added |
+| QUEUE | REMOVE | DETAILS_NONE | | Removing configured queue |
+| QUEUE | REMOVE | QUEUE_DYNAMIC | | Removing dynamic queue |
+| QUEUE | REMOVE | QUEUE_APP | ApplicationID | Application removed |
+| QUEUE | SET | QUEUE_MAX | | Max resource changed |
+| QUEUE | SET | QUEUE_GUARANTEED | | Guaranteed resource changed |
+| QUEUE | SET | QUEUE_TYPE | | Queue type change(parent/leaf) |
+
+### Shim events
+
+Shim events are a type of REQUEST and the application ID is set.
+Right now we send only two types of events to the shim, both of which are originated from the application:
+
+- Insufficient resources in the queue for a request
+- Real allocation is larger than placeholder
+
+| Event type | Change type | Change details | Reference type | Notes |
+|------------|-------------|----------------|----------------|-----------------------|
+| REQUEST | NONE | DETAILS_NONE | ApplicationID | New application added |
+| REQUEST | NONE | DETAILS_NONE | ApplicationID | Successful allocation |
+
+## Configuration
+
+As part of the design we anticipate the following configuration to be added for the event storage and retrieval.
+All settings will have the shared prefix: *service.event*.
+All the mentioned settings will require a restart of the scheduler for updates to take effect.
+
+The following settings will be added to the system:
+
+A flag to enable or disable sending request events to the K8shim.
+This externalises the current setting that needs a recompile of the code.
+The default mirrors the current default in the code.
+
+- Name: **requestEventsEnabled**
+- Allowed values: all boolean values will be converted into a boolean using ParseBool as defined in the `strconv` package[^1].
+- Default value: false
+
+A flag to enable or disable event collection in the system.
+The default is enabled for the tracking events.
+
+- Name: **trackingEventsEnabled**
+- Allowed values: all boolean values will be converted into a boolean using ParseBool as defined in the `strconv` package.
+- Default value: true
+
+Size of the store for events that will be sent to the shim, request events.
+Note that setting a size of 0 effectively turns off the request event system.
+
+- Name: **requestStoreCapacity**
+- Allowed values: integer value that can be converted into a 32-bit integer using ParseUint as defined in the `strconv` package[^2].
+- Default value: 1,000
+
+The number of events to store in the ring buffer.
+This number defines the size of the ring buffer and thus the memory impact the event storage will have on the scheduler.
+Note that setting a size of 0 effectively turns off the event collection system.
+
+- Name: **ringBufferCapacity**
+- Allowed values: integer value that can be converted into a 32-bit integer using ParseUint as defined in the `strconv` package
+- Default value: 100,000
+
+The maximum number of events to return in one REST response.
+The maximum that could be returned is limited to the ring buffer capacity.
+However, preparing a large response the size of the whole ring buffer could cause large peaks in the schedulers memory usage.
+This setting should be used to prevent these peaks.
+
+- Name: **RESTResponseSize**
+- Allowed values: integer value that can be converted into a 32-bit integer using ParseUint as defined in the `strconv` package
+- Default value: 10,000
+
+## Performance considerations
+
+### Memory usage of historical elements
+
+We need to estimate how many elements we can expect to be generated with different pod churn rates on a daily basis.
+Queue and node events are much rarer, so they are not accounted for.
+We also ignore potential failures and scheduling errors.
+
+An application undergoes state transitions, so the following events are generated:
+- Add new application
+- State change: New
+- State change: Accepted
+- State change: Running
+- State change: Completing
+- State change: Completed
+- Allocation changed
+
+Pod (request) events:
+- Incoming allocation request
+- Successful allocation on a node
+
+Pod (request) events with gang scheduling:
+- Incoming resource request
+- Successful allocation on a node
+- Replace event for placeholder
+
+| Number of pods | Number of apps | Pods per app | Job type | Events per app | Events per pod | ∑ apps | ∑ pods | ∑ events |
+|----------------|----------------|--------------|----------|----------------|----------------|-------------------|-----------------|----------|
+| 150k | 1500 | 100 | normal | 200 + 7 | 2 | 310k (1500 * 207) | 300k (2 * 150k) | 610k |
+| 150k | 1500 | 100 | gang | 300 + 7 | 3 | 460k (1500 * 307) | 450k (3 * 150k) | 910k |
+| 300k | 3000 | 100 | normal | 200 + 7 | 2 | 621k (3000 * 207) | 600k (2 * 300k) | 1,2M |
+| 300k | 3000 | 100 | gang | 300 + 7 | 3 | 921k (3000 * 307) | 900k (3 * 300k) | 1,8M |
+
+On a busy cluster with 150,000 - 300,000 pods per day, we can expect the number of events generated to be around 600k - 1,8M (depending on the scheduling style).
+
+If we want to retain the events for 5 days, we need to store 9 million events in the worst case and 3 million in the best case.
+
+With 9 million objects in the memory, it is also critical to estimate how much extra memory usage is expected from YuniKorn with different kinds of data structures:
+
+- slice of pointers to si.EventRecord objects `[]*si.EventRecord`
+- slice of si.EventRecord objects `[]si.EventRecord`
+- slice of a memory-optimised data type `[]internalRecord`
+
+As the name suggests, internalRecord type would only be available inside the ring buffer.
+It's used together with string interning to save memory caused by repeated strings like IDs (application ID, ask ID, allocation ID, etc.).
+There are various interning libraries in Go, but removing elements is not possible from them.
+We need to create our own, so we can track the number of references per string.
+
+The following table summarizes memory usage and GC time, depending on how we store the events.
+The test was executed multiple times to get averages and ranges.
+The slice was created once, then an index variable was increased for each record, so the built-in `append()` function was not used.
+
+The results clearly indicate two things:
+
+- having millions of pointers has negative performance on both the memory usage and GC load
+- batch workloads inherently have a lot of repeated strings, so interning makes a noticeable difference
+
+The GC detects pointers, since it has to follow them to walk the object graph to identify unreferenced objects.
+If elements in the slice are pointers, then those allocations are scattered all around the heap (bad locality) with header information added by the allocator.
+This reduces memory access times and increases the amount of the data that is allocated.
+
+In the test, the assumption was 100 pods per application.
+In real life, this value might be different, most likely it's lower.
+The lower the value, the smaller the advantage of the interning - on the extreme, every application consists of only one pod.
+But even with only 30 pods per application, it has value - more modest, 20-30% reductions are not negligible when storing millions of elements.
+
+| Object type | Number of elements | Increased memory usage\* | GC time\*\* |
+|-------------------|--------------------|--------------------------|-------------|
+| []*si.EventRecord | 3M | 805 MiB | 85-105 ms |
+| []si.EventRecord | 3M | 437 MiB | 41-55 ms |
+| []internalRecord | 3M | 211 MiB | 5-16 ms |
+| []*si.EventRecord | 6M | 1595 MiB | 160-200 ms |
+| []si.EventRecord | 6M | 856 MiB | 75-110 ms |
+| []internalRecord | 6M | 404 MiB | 10-30 ms |
+| []*si.EventRecord | 9M | 2380 MiB | 270-320 ms |
+| []si.EventRecord | 9M | 1280 MiB | 116-150 ms |
+| []internalRecord | 9M | 593 MiB | 16-33 ms |
+
+*\* "Sys" metrics returned by the Go runtime*
+
+*\*\* measured by the execution time of `runtime.GC()`*
+
+### Handling slow and rogue clients during streaming
+
+It's possible to overwhelm YuniKorn with streaming or execute a DoS attack.
+
+The first thing we need to address is a rogue client which simply stops reading on the receiver side from the TCP socket.
+Eventually writes will block in YuniKorn.
+If we use channels and per-client goroutines, this should not be an issue - we can always check if the buffer is full using the built-in len() function.
+If this happens, we can just drop the connection.
+In some situations, the buffer is expected to be full, for example, when sending history.
+It's always faster to send out a lot of existing objects, so we can expect the buffer to get full at times.
+This can be solved by repeatedly checking the rate of writes - if it falls below a certain threshold, then again, we just drop the connection with an error message.
+
+We can also limit the number of concurrent streaming connections and historical objects to send, e.g. we simply won't serve requests which would result in sending millions of elements.
+
+Note that it's still possible to deny the service from legitimate users because YuniKorn lacks authentication on the REST interface.
+This will be solved in the future.
+
+### Approaches to streaming
+
+When a client requests a stream of events, it can define how many past events it wants to receive.
+This is necessary because when YuniKorn starts up, it's not possible to connect immediately.
+It can be important for certain users to save the state of YuniKorn on an external storage completely from the start.
+
+Sending the historical events must be separated from the new events that are constantly occurring in the system, that is, old and current events must not interleave.
+For example: we want a stream of old and new events, this means:
+
+- We're currently at *event~n~*
+- Send out events from *event~0~* all the way up to *event~n~*
+- In the meantime, *k* number of events have been generated
+- Therefore, send events up to event*~n+k~* until we're caught up completely
+
+#### **Send history first, then stream new events**
+
+The first approach is to retrieve the objects from the ring buffer and start to send them on a channel to the receiver.
+There is a separate buffered channel for new events with capacity *localBufferSize*.
+As soon as we finish sending the history, we switch to this channel and relay events from it towards the client.
+
+We can express this with the following pseudocode:
+```go
+consumerEventLoop(elementCount) {
+ local = initLocalEventChannel(localBufferSize)
+ // send history
+ history := ringbuffer.getHistory(elementCount)
+ foreach h : history {
+ receiver <- h
+ }
+ // bridge new events
+ foreach e : <-local {
+ receiver <- e
+ }
+}
+```
+
+The publisher logic is:
+```go
+publishEvent(event) {
+ foreach c : localChannels {
+ if len(c) == localBufferSize { // local buffer full?
+ closeChan(receiver)
+ closeChan(local)
+ receiverId = getReceiver(c)
+ abort("slow receiver: " + receiverId)
+ }
+ c <- event
+ }
+}
+```
+
+Each receiver has its own goroutine.
+That is, 10 clients means 10 different `consumerEventLoop()` calls running on separate goroutines.
+
+To detect slow clients, we simply check the value of `len(c)` whenever we submit the new event.
+If this equals the capacity of the buffer, we drop the receiver and close the connection.
+
+It is possible to send events directly to the receiver channel and skip `c` on the publisher side which is local inside consumerEventLoop.
+This requires a simple state machine and a temporary slice of new events, since we are not allowed to send directly as long as the history is being transmitted.
+Overall, this is more complicated than the one described above.
+
+The simplicity of this method is notable, which is the main advantage: reasoning about its correctness is trivial.
+We do not need complicated test code and back-and-forth review cycles to ensure that it indeed results in the correct order of events.
+
+The biggest question is what happens if channel local becomes full.
+Let's say that localBufferSize is 1000.
+On a busy cluster with 100 events / second (busiest period), it takes 10 seconds to fill it up completely.
+However, nothing stops us from setting the capacity of the buffer to a higher value.
+If we bump it up to 10000, the client is expected to read the history in 100 seconds.
+On a decent network, this amount of time should be enough to read the event history, even if it means 100 000 entries from the ring buffer.
+If a single si.EventRecord results in 300 bytes of JSON text, then it is around 30MiB of data.
+Even if we take serialisation into account, this will not (can not) take minutes.
+
+The biggest disadvantage of this method is when a user requests a lot of elements from the history.
+For example, someone wants to retrieve the last 1M elements.
+In this case, we need to allocate a slice for 1M entries.
+Based on the measurements, this will generate a sudden spike in the live heap size, around 145-150 MiB. If people repeatedly do this, the memory allocation of YuniKorn can increase very quickly, possibly resulting in a termination of the YuniKorn pod by the kubelet.
+
+#### **Stream new events directly from ring buffer**
+
+Instead of notifying receivers directly, we just insert the data into the ring buffer.
+For every client, we create a cursor which points to a given position inside the buffer.
+If the pointer is behind the tail, it just keeps reading and going forward until it reaches the current tail.
+
+Although this method sounds simple on the surface, the implementation is complicated:
+
+- When sending the history, we can't rely on `len(buf)` checking.
+It is because when we are sending the history, we are always very close to the capacity.
+Unlike the previous solution, we are processing elements from the buffer directly, and we do not bridge two channels.
+If we do that, we just add further complications.
+Therefore, we have to calculate the sending rate, at least while sending past objects.
+If the pointer is not making progress (or just very slow), then the tail of the ring buffer will catch up to it, but it can take a very long time.
+- In case of multiple cursors, we have to maintain which one is the closest to the tail.
+If we don't do that, we have to check every cursor when adding elements.
+We cannot wait for a slow reader to jump to the next element, so we have to invalidate the cursor, then have to calculate which is the nearest to tail again.
+- Whenever we reach the last element, we have to block and utilize the well-known wait-notify pattern (`Cond.Broadcast()` and `Cond.Wait()` in Go).
+- The overall sending logic is much more complicated: keep reading batches from the ring buffer until we reach the last element, then switch to wait-notify and block.
+
+The advantage of this method is that we don't need large channel buffers to make sure that they don't get filled while sending historical records.
+If the reader is fast enough, we just keep marshalling `si.EventRecord` objects and bump the pointer to the next element.
+
+[^1]: https://pkg.go.dev/strconv#ParseBool
+
+[^2]: https://pkg.go.dev/strconv#ParseUint
diff --git a/versioned_docs/version-1.5.1/design/interface_message_simplification.md b/versioned_docs/version-1.5.1/design/interface_message_simplification.md
new file mode 100644
index 0000000000..b6766ced48
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/interface_message_simplification.md
@@ -0,0 +1,309 @@
+---
+id: interface_message_simplification
+title: Simplifying Interface Messages
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+
+# Simplifying Interface Messages and Breaking Shim build dependency on Core
+
+# Proposal
+This document describes a) complexity hidden behind existing Interface messages and
+explains the newly defined SI messages and its dependent changes on Core and Shim.
+b). Breaking Shim build dependency on Core.
+
+## Goals
+The goal is to provide the same functionality before and after the change.
+- unit tests before and after the merge must all pass.
+- Smoke tests defined in the core should all pass without major changes definition.
+- End-to-end tests that are part of the shim code must all pass without changes.
+## Background
+The current interface allows us to only send one message between a shim and the core. This provides us with a really simple way of interactions definition.
+
+The complexity is however hidden in the message itself. Every message serves multiple purposes and when the message is received the core and shim need to unpack it and process each part separately and for certain parts in a real specific order.
+Because the message serves a number of purposes it has a large overhead. This might not show up in the code directly as the heavy lifting is done in the generated code. It will show up in the amount of data as a message, even if it does not have all fields, still needs to be encoded in a way that it unpacks correctly on the other side.
+
+## Simplifying Interface Messages
+
+Proposal is to split the one large message into 3 separate messages - one for each entity:
+
+- Allocations
+- Applications
+- Nodes
+
+### API Interface Changes
+
+```
+package api
+
+import "github.com/apache/incubator-yunikorn-scheduler-interface/lib/go/si"
+
+type SchedulerAPI interface {
+ // Register a new RM, if it is a reconnect from previous RM, cleanup
+ // all in-memory data and resync with RM.
+ RegisterResourceManager(request *si.RegisterResourceManagerRequest, callback ResourceManagerCallback) (*si.RegisterResourceManagerResponse, error)
+
+ // Update Allocation status
+ UpdateAllocation(request *si.AllocationRequest) error
+
+ // Update Application status
+ UpdateApplication(request *si.ApplicationRequest) error
+
+ // Update Node status
+ UpdateNode(request *si.NodeRequest) error
+
+ // Notify scheduler to reload configuration and hot-refresh in-memory state based on configuration changes
+ UpdateConfiguration(clusterID string) error
+}
+
+// RM side needs to implement this API
+type ResourceManagerCallback interface {
+
+ //Receive Allocation Update Response
+ UpdateAllocation(response *si.AllocationResponse) error
+
+ //Receive Application Update Response
+ UpdateApplication(response *si.ApplicationResponse) error
+
+ //Receive Node update Response
+ UpdateNode(response *si.NodeResponse) error
+
+ // Run a certain set of predicate functions to determine if a proposed allocation
+ // can be allocated onto a node.
+ Predicates(args *si.PredicatesArgs) error
+
+ // RM side implements this API when it can provide plugin for reconciling
+ // Re-sync scheduler cache can sync some in-cache (yunikorn-core side) state changes
+ // to scheduler cache (shim-side), such as assumed allocations.
+ ReSyncSchedulerCache(args *si.ReSyncSchedulerCacheArgs) error
+
+ // This plugin is responsible for transmitting events to the shim side.
+ // Events can be further exposed from the shim.
+ SendEvent(events []*si.EventRecord)
+
+ // Scheduler core can update container scheduling state to the RM,
+ // the shim side can determine what to do incorporate with the scheduling state
+ // update container scheduling state to the shim side
+ // this might be called even the container scheduling state is unchanged
+ // the shim side cannot assume to only receive updates on state changes
+ // the shim side implementation must be thread safe
+ UpdateContainerSchedulingState(request *si.UpdateContainerSchedulingStateRequest)
+
+ // Update configuration
+ UpdateConfiguration(args *si.UpdateConfigurationRequest) *si.UpdateConfigurationResponse
+}
+```
+
+### Interface Changes to replace UpdateRequest
+
+UpdateRequest would be divided into below messages:
+
+#### AllocationRequest
+```
+message AllocationRequest {
+ repeated AllocationAsk asks = 1;
+ AllocationReleasesRequest releases = 2;
+ string rmID = 3;
+}
+```
+#### ApplicationRequest
+```
+message ApplicationRequest {
+ repeated AddApplicationRequest new = 1;
+ repeated RemoveApplicationRequest remove = 2;
+ string rmID = 3;
+}
+```
+#### NodeRequest
+```
+message NodeRequest {
+ repeated NodeInfo nodes = 1;
+ string rmID = 2;
+}
+```
+### Merging Create and Update NodeInfo into Single NodeInfo
+```
+message NodeInfo {
+ enum ActionFromRM {
+ CREATE = 0;
+ UPDATE = 1;
+ DRAIN = 2;
+ SCHEDULABLE = 3;
+ DECOMISSION = 4;
+ }
+
+ string nodeID = 1;
+ ActionFromRM action = 2;
+ map<string, string> attributes = 3;
+ Resource schedulableResource = 4;
+ Resource occupiedResource = 5;
+ repeated Allocation existingAllocations = 6;
+}
+```
+
+### Event Changes to replace UpdateRequest
+
+RMUpdateRequestEvent would be replaced by following events:
+
+- RMUpdateAllocationEvent
+- RMUpdateApplicationEvent
+- RMUpdateNodeEvent
+
+### Interface Changes to replace UpdateResponse
+
+UpdateResponse would be divided into below messages:
+
+#### AllocationResponse
+```
+message AllocationResponse {
+ repeated Allocation new = 1;
+ repeated AllocationRelease released = 2;
+ repeated AllocationAskRelease releasedAsks =3;
+ repeated RejectedAllocationAsk rejected = 4;
+}
+```
+#### ApplicationResponse
+```
+message ApplicationResponse {
+ repeated RejectedApplication rejected = 1;
+ repeated AcceptedApplication accepted = 2;
+ repeated UpdatedApplication updated = 3;
+}
+```
+#### NodeResponse
+```
+message NodeResponse {
+ repeated RejectedNode rejected = 1;
+ repeated AcceptedNode accepted = 2;
+}
+```
+
+### Event Changes for UpdateResponse
+
+Scheduler/Context.go from Core already triggers an event for each entity separately and rmproxy.go is the one which handles all these events and packs it under single *si.UpdateResponse and eventually sends to shim through scheduler_callback#RecvUpdateResponse. With the above API interface change, rmproxy.go would use appropriate callback method to send response to shim. With this separate callback approach, each entity response would be handled separately in shim.
+
+## Detailed Flow Analysis
+
+### Add/Delete Allocations
+
+The RM (shim) sends a simplified AllocationRequest as described above. This message is wrapped by the RM proxy and forwarded to the cache for processing. The RM can request an allocation to be added or removed.
+
+```
+1. Shim sends a simplified AllocationRequest to core through SchedulerAPI.UpdateAllocation
+2. RMProxy sends rmevent.RMUpdateAllocationEvent to scheduler
+3. On receiving the above event, scheduler calls context.handleRMUpdateAllocationEvent to do the
+ following:
+ 3.1: processAsks
+ 2.1.2: Process each request.Asks ask of AllocationRequest request and adds to the application
+ 2.1.2: In case of rejection, triggers RMRejectedAllocationAskEvent with
+ all asks which has been rejected
+ 2.1.2: On receiving RMRejectedAllocationAskEvent, RMProxy.processUpdatePartitionConfigsEvent
+ process the event, creates a AllocationResponse using RMRejectedAllocationAskEvent
+ attributes and send to shim through UpdateAllocation callback method
+ 3.2: processAskReleases
+ 2.2.1: Process each request.Releases.AllocationAsksToRelease ask release of AllocationRequest request
+ and removes from the application
+ 3.3: processAllocationReleases
+ 3.3.1: Process each request.Releases.AllocationRelease allocation release of AllocationRequest
+ request and removes from the application
+ 3.3.2: Collect all above exact released allocations and triggers RMReleaseAllocationEvent with all allocations needs to be released
+ 3.3.3: On receiving RMReleaseAllocationEvent, RMProxy.processRMReleaseAllocationEvent
+ prcoess the event, creates a AllocationResponse using RMReleaseAllocationEvent
+ attributes and send to shim through UpdateAllocation callback method
+ 3.3.4: Collect all above confirmed (placeholder swap & preemption) allocations
+ and send to shim through two ways
+ a). Wraps confirmed allocations as AssumedAllocation
+ and send to shim ReSyncSchedulerCache callback plugin
+ b). Wraps confirmed allocations as Allocation and triggers
+ RMNewAllocationsEvent. On receiving RMNewAllocationsEvent,
+ RMProxy.processAllocationUpdateEvent process the event, creates a
+ AllocationResponse using RMNewAllocationsEvent attributes and send to shim
+ through UpdateAllocation callback method
+```
+
+### Add/Delete Applications
+
+The RM (shim) sends a simplified ApplicationRequest as described above. This message is wrapped by the RM proxy and forwarded to the cache for processing. The RM can request an application to be added or removed.
+
+```
+1. Shim sends a simplified ApplicationRequest to core through SchedulerAPI.UpdateApplication
+2. RMProxy sends rmevent.RMUpdateApplicationEvent to scheduler
+3. On receiving the above event, scheduler calls context.handleRMUpdateApplicationEvent to do the
+ following:
+ 3.1: Add new apps to the partition.
+ 3.1.2: Wraps AcceptedApps and RejectedApps (if any) as part of RMApplicationUpdateEvent
+ and fires the same
+ 3.1.2: On receiving RMApplicationUpdateEvent, RMProxy.processApplicationUpdateEvent
+ process the event, creates a ApplicationResponse using RMApplicationUpdateEvent
+ attributes and send to shim through UpdateApplication callback method
+ 3.2 Remove apps from the partition.
+ 3.2.1: Collect all allocations belongs to the removed app and triggers
+ RMReleaseAllocationEvent with all allocations needs to be released
+ 3.2.2: On receiving RMReleaseAllocationEvent, RMProxy.processRMReleaseAllocationEvent
+ prcoess the event, creates a AllocationResponse using RMReleaseAllocationEvent
+ attributes and send to shim through UpdateAllocation callback method
+```
+
+### Add/Delete Nodes
+
+The RM (shim) sends a simplified NodeRequest as described above. This message is wrapped by the RM proxy and forwarded to the cache for processing. The RM can request an node to be added or removed.
+
+```
+1. Shim sends a simplified NodeRequest to core through SchedulerAPI.UpdateNode
+2. RMProxy sends rmevent.RMUpdateNodeEvent to scheduler
+3. On receiving the above event, scheduler calls context.handleRMUpdateNodeEvent to do the
+ following:
+ 3.1: Add new node to the partition.
+ 3.1.2: Wraps AcceptedNodes and RejectedNodes (if any) as part of RMNodeUpdateEvent
+ and fires the same
+ 3.1.2: On receiving RMNodeUpdateEvent, RMProxy.processRMNodeUpdateEvent
+ process the event, creates a NodeResponse using RMNodeUpdateEvent
+ attributes and send to shim through UpdateNode callback method
+ 3.2: Update node
+ 3.2.1 Update the partition resource
+ 3.3: Drain node
+ 3.3.1 Ensures node is not schedulable
+ 3.4: Decommissioning (Remove) node from the partition.
+ 3.4.1: Ensures node is not schedulable
+ 3.4.2: Collect all above exact released allocations from that node and triggers
+ RMReleaseAllocationEvent with all allocations needs to be released
+ 3.4.3: On receiving RMReleaseAllocationEvent,
+ RMProxy.processRMReleaseAllocationEvent process the event, creates a
+ AllocationResponse using RMReleaseAllocationEvent attributes and
+ send to shim through UpdateAllocation callback method
+ 3.4.4: Collect all above confirmed (placeholder swap & preemption) from that node
+ allocations and send to shim through two ways
+ a). Wraps confirmed allocations as AssumedAllocation and send to shim
+ through ReSyncSchedulerCache callback plugin
+ b). Wraps confirmed allocations as Allocation and triggers RMNewAllocationsEvent.
+ On receiving RMNewAllocationsEvent, RMProxy.processAllocationUpdateEvent
+ process the event, creates a AllocationResponse using RMNewAllocationsEvent
+ attributes and send to shim through UpdateAllocation callback method
+```
+
+## Breaking the Shim build dependency on Core
+
+Planned for different phases.
+
+### Phase 1
+Moved all plugins from core to appropriate place in SI under ResourceManagerCallback,
+a single common interface.
+
+### Phase 2
+Please refer https://issues.apache.org/jira/browse/YUNIKORN-930 for more details
diff --git a/versioned_docs/version-1.5.1/design/preemption.md b/versioned_docs/version-1.5.1/design/preemption.md
new file mode 100644
index 0000000000..0106bddc7a
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/preemption.md
@@ -0,0 +1,581 @@
+---
+id: preemption
+title: Preemption
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+# Preemption Design
+
+## Introduction
+
+This design is created to replace the preemption code that has been part of
+YuniKorn for a long time.
+
+The original code was never fully tested. After the core cache removal as part
+of [YUNIKORN-317](https://issues.apache.org/jira/browse/YUNIKORN-317) the
+preemption code was taken out of the scheduling path. The only changes that
+have gone into the preemption code were compilation fixes. As part of the
+preparation for the new preemption design the old code was removed in
+[YUNIKORN-1269](https://issues.apache.org/jira/browse/YUNIKORN-1269).
+
+Preemption of workloads is a feature used by almost all schedulers to help
+critical workloads run. The criticality can be based on the fact that the
+system cannot work without it, i.e. DaemonSets on K8s, or based on some
+other factors like SLA or priority. Preemption for DaemonSet pods was
+implemented as part of
+[YUNIKORN-1085](https://issues.apache.org/jira/browse/YUNIKORN-1085). This
+design thus focuses on preemption for all other workloads.
+
+## Goals
+
+* Re-use existing Kubernetes objects
+* Implement inter-queue preemption
+* Priority knowledge inside the core scheduler to support preemption
+
+## Non Goals
+
+* Intra-queue preemption
+* Cross node preemption
+* Full priority support
+
+## Preemption and Priorities
+
+### Kubernetes
+
+All pods for the entire cluster, across namespaces, are sorted based on
+priority for scheduling. A higher-priority will get scheduled before a
+lower-priority pod. Preemption within the Kubernetes scheduler is also
+based solely on the priority of the pod that is being scheduled. The
+full documentation can be found
+[here](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/).
+
+The simple rule that governs both scheduling order and preemption behavior is
+that a pod with a higher priority is more important than a pod with a lower
+priority. This crosses namespaces. There are no boundaries or limitations
+that can be set. Pods from a namespace with a higher priority are scheduled
+earlier and could trigger the preemption of a pod from another namespace.
+
+Priority of a pod is defined in the pod specification. Priority is a number,
+but is referenced in the pod specification via a PriorityClass object. The
+object defines a named priority and maps it to a value. In more recent
+versions (Kubernetes 1.24 and later), the PriorityClass can also contain a
+preemption policy for the scheduling phase. This preemptionPolicy only allows
+the pod to opt-out from preempting other lower priority pods during the
+scheduling phase. It cannot, and is not designed to, alter preemption behavior
+while the pod is running.
+
+The same pod priority is also used during the pod eviction by the kubelet on
+the node. For example, if the node has been overloaded and runs out of memory
+or CPU, the node can evict pods. These evictions help to keep the node stable.
+Pods considered for eviction are ranked on priority among some other factors.
+
+### YuniKorn
+
+The behavior in Kubernetes is all focused on the scheduling cycle. When
+looking at preemption from the standpoint of a batch or data processing
+workload, we need to take into account the possibility to opt-out while
+running. This option does not exist in the current configuration for workloads.
+
+The simple approach for preemption fits in with the service type workloads
+that Kubernetes is designed for. Services handle the scale up and or scale
+down that could be triggered by preempting one of the pods for that service.
+
+This is not the case when we look at a Spark job as an example. The driver pod
+is the manager of the other pods that belong to the same job. When the driver
+gets preempted the whole job is impacted. Preempting a driver has a follow-on
+effect that could cause multiple pods to be removed from the cluster. For a
+large job that could mean hundreds of pods. The impact is far larger than a
+single pod. Preempting a manager or originator pod should be prevented as much
+as possible.
+
+The other case which we need to account for is an interactive session that
+runs on the cluster. This interactive session, like a python notebook, is a
+one-off instance and restarts have a wider impact. Preempting an instance
+like a notebook is not a desirable outcome either.
+
+Building in all the smarts to know which pods to avoid is difficult, or even
+impossible. For all these cases the option to allow a “do not preempt me” flag
+on the pod would be required. The design will discuss this option in the Pod
+specification.
+
+### Priority limitations
+
+PriorityClass objects which define the priority in a Kubernetes cluster are
+cluster-wide objects. The cluster administrator defines the objects and thus
+the supported priorities.
+
+There are no limitations on the priority that can be set on a pod. When a pod
+is submitted the only check performed is that the PriorityClass specified as
+part of the pod specification is defined. Any rogue user can create a pod with
+the highest defined priority.
+
+Limiting priorities is possible but requires an API server flag which is
+difficult to support in public cloud environments. We cannot rely on the option
+to be available in a cluster scheduled by YuniKorn.
+
+## The Laws of Preemption
+
+Prior knowledge of implementing preemption in the Apache YARN scheduler has
+been used to create this design. It forms the basis for the "laws of preemption".
+These laws, or rules, describe overall preemption behavior.
+
+Preemption for YuniKorn is based on the hierarchical queue model and guaranteed
+resources assigned to a queue.
+
+Preemption is used to get the usage of a queue to at least its guaranteed
+resource capacity. It cannot guarantee that every queue will get its
+guaranteed resource capacity. There are cases that make it impossible to get to
+that state. Preemption therefore does not guarantee a final optimal state can
+or will be reached. The end state of the queues will normally thus be a usage
+that is approximately the guaranteed resource capacity.
+
+An example case that makes it impossible to reach the optimal state is when the
+sum of the guaranteed resources for a group of queues is larger than the current
+resources available in the cluster. It is impossible to get all queues their
+guaranteed resources. More cases like this exist in a hierarchy with maximum
+quotas set at a certain point in the hierarchy preventing the optimal state from
+being reached.
+
+First an overview of the laws, then the explanation and rationale behind each
+of the rules:
+
+1. Preemption policies are strong suggestions, not guarantees
+2. Preemption can never leave a queue lower than its guaranteed capacity
+3. A task cannot preempt other tasks in the same application
+4. A task cannot trigger preemption unless its queue is under its guaranteed
+ capacity
+5. A task cannot be preempted unless its queue is over its guaranteed capacity
+6. A task can only preempt a task with lower or equal priority
+7. A task cannot preempt tasks outside its preemption fence (one-way constraint)
+
+Most of the rules given around tasks and queues are there to prevent a
+preemption storm or loop. Preempting a task that could trigger a follow-up
+preemption should be avoided. The rule described should warrant against that.
+Breaking the rules could lead to unpredictable behavior either during
+scheduling or application runs.
+
+### Preemption policies are strong suggestions, not guarantees
+
+A pod is able to request not to be preempted. However, this is not a guarantee.
+There could be circumstances that the scheduler might still preempt the pod.
+This will be a last resort action if no other solution is available. This means
+that the preemption policy is a strong suggestion, not a guarantee.
+
+The example use case that could break the rule and preempt a pod that opted out
+of preemption is the DaemonSet case. The DaemonSet pod must run on the specified
+node. If all pods that are running on that node are either other DaemonSet pods
+or pods that have opted out of preemption there is no other choice. A pod that
+opted out of preemption will be preempted.
+
+### Preemption can never leave a queue lower than its guaranteed capacity
+
+Guaranteed resource capacities configured on a queue are the optimal resources
+for the queue from a preemption point of view. Since preemption attempts to get
+a queue at least its guaranteed resources, allowing preemption to reduce a queue
+below its guaranteed resource capacity would trigger a further round of
+preemption.
+
+When a task is considered for preemption the removal of the task from the
+queue's usage must not leave the queue below its guaranteed resource capacity.
+If the removal of the task would drop the queue below its guarantee that task
+cannot be a candidate.
+
+This could mean that a queue, even when it is above its guarantee, still does
+not allow preempting any of the tasks in that queue.
+
+### A task cannot preempt other tasks in the same application
+
+As part of the current design we are only implementing inter-queue preemption.
+This rule does not apply for inter-queue preemption. It is documented here to
+make extending to intra-queue preemption possible without a rule change.
+
+When a task is considered for preemption the task that triggered the preemption
+cannot be from the same application.
+
+### A task cannot trigger preemption unless its queue is under its guaranteed capacity
+
+Preemption attempts to get a queue at least its guaranteed resources. Allowing
+preemption for a task in a queue that is already over its guaranteed resource
+capacity does not help reaching that goal.
+
+Preemption is not the tool to get a queue to its full maximum resource
+capacity. For that reason no tasks in a queue are allowed to trigger
+preemption in this case. Normal scheduling will handle the assignment of
+resources for queues above their guaranteed capacity.
+
+### A task cannot be preempted unless its queue is over its guaranteed capacity
+
+This is an extension to the rule that prevents a queue from going below its
+guaranteed resource capacity. If a queue is already below its guaranteed
+capacity, no tasks running in that queue can be considered for preemption.
+The queue can already trigger preemption of tasks from other queues. Allowing
+preemption to remove more resources from the queue would cause the queue to be
+even further from its ideal usage.
+
+### A task can only preempt a task with lower or equal priority
+
+This rule is linked to the default preemption behavior of Kubernetes. There is
+however a slight difference when compared to the Kubernetes rule. Kubernetes
+only allows pods to preempt other pods with a lower priority, not with equal
+priority.
+
+Preemption in YuniKorn has a slightly broader application as we also use it to
+allow a queue to use its guaranteed resource capacity. The equality has been
+added to allow pods from the same priority running in two different queues to
+preempt each other. Without that equality allowance on priorities that
+redistribution would be slow and difficult.
+
+### A task cannot preempt tasks outside its preemption fence
+
+This last rule is a specific multi-tenancy rule. YuniKorn allows setting up a
+preemption fence on a queue that will limit the possible tasks that can be
+considered for preemption. A task can only preempt another task if it runs on
+a queue that is located inside the preemption fence. More on preemption
+fencing below.
+
+## Preemption Fence
+
+In a cluster that runs workloads for multiple tenants preemption should not be
+allowed to preempt a workload from one tenant and give the resources to
+another. A tenant in this case does not have to be a company, it could be a
+division or business group within the same company. Tenants map to the queue
+hierarchy. The queue hierarchy can thus cross tenant boundaries.
+
+To translate this to a YuniKorn centric view: confining preemption to assess
+only a part of the queue hierarchy should be possible. For this we want to
+introduce a preemption fence. The preemption fence is a setting on the queue
+which stops preemption from looking at queues outside the fence boundary.
+
+The preemption fence can be set on any queue at any point in the hierarchy.
+It blocks traversing up the queue hierarchy when looking for a victim to
+preempt. The fence is a unidirectional fence. It prevents going out (i.e.
+higher up the queue hierarchy), but does not prevent coming in (or down) the
+queue hierarchy. The below diagram shows an example hierarchy with preemption
+fencing applied:
+
+
+
+The example has three preemption fences defined at different levels. The
+arrows describe the preemption flows.
+
+First look at `queue A`.
+A fence is defined on the leaf queue which means that preemption is fenced to
+that queue. This would limit tasks to find a preemption victim within the
+queue. This in effect turns off inter-queue preemption for the leaf
+`queue A`. This would be a case for intra-queue preemption only. Note that in
+the current design intra-queue preemption is excluded, and we will not
+implement this use case. However, we do allow the configuration to be set.
+
+The second fence is set up at the `tenant 1` level. This fence has an impact
+on the task running in `queue B`. The tasks in `queue B` can find a victim
+to preempt in all queues below the `tenant 1` queue (shown in the diagram with
+the _yellow_ arrows).
+
+A similar setup is created for the `tenant 2` queue. However, in this case
+there can be preemption from `queue 1` to `queue 2` and vice-versa (shown in
+the diagram with the _blue_ arrows).
+
+None of the tasks running in the queues below `tenant 1` or `tenant 2` can
+preempt a task from the `system` queue. Tasks in the `system` queue are not
+fenced. That means that a task in the `system` queue can check any queue in
+the hierarchy (shown in the diagram as the _red_ arrows).
+
+## Preemption Logic
+
+For preemption to be triggered by the default Kubernetes scheduler, the pod
+must fit within any namespace quota and no node can be found to allocate the
+pod on. Preemption tries to find a node to run the pod on and is focused on
+nodes only. All pods on a node with a lower priority are considered candidates
+for preemption. The scheduler reduces the candidates to a victim list just
+large enough to fit the new pod.
+
+The scheduler tracks the node undergoing preemption in the pod that triggered
+the preemption. It does not guarantee, for a number of reasons, that the pod
+that triggered the preemption also gets scheduled on the node the pods were
+preempted from.
+
+From the YuniKorn side we already have a node focused preemption for DaemonSet
+pods. A pod from a DaemonSet must run on a specific node. Preemption is
+focused on making space on that node. YuniKorn guarantees that the pod will
+run on that node and nothing else will be scheduled on that node until the
+DaemonSet pod is scheduled.
+
+Generic preemption in YuniKorn is linked to a queue that is under its
+guaranteed quota (**rule 4**). The preemption is only triggered after a delay.
+A request must have been pending for at least the preemption delay before
+preemption will be considered for it. Instead of looking at a node the first
+step is to find victims based on the queue that is over its guaranteed quota
+(**rule 5**). If no queues can be found within the preemption fence
+(**rule 7**), preemption immediately stops.
+
+The next step is to build a list of victims. This could be from one or
+multiple queues. Any pod with a priority higher (**rule 6**) than the pod
+triggering the preemption is discarded. The next step is to discard any pod
+with resources larger than the over guarantee portion (**rule 2**). Proximity
+of the victim queue in the hierarchy to the originating queue that triggers
+the preemption might be considered when we decide which victims to preempt.
+Details will be worked out during the implementation phase.
+
+Cross node preemption is not supported. Any set of victims that get preempted
+must run on the same node. The victims are split into groups per node. A group
+of victims, combined with the allocatable space on the node, will decide if
+the group is a viable group. Checks must be run to make sure that the pod
+will fit the node before starting the preemption. If more than one group is
+viable a choice will need to be made. The logic for choosing a group is an
+implementation detail.
+
+After choosing a group and indirectly a node, the node is reserved for the
+pod. Preemption for the victim list will be triggered and on completion the
+pod will be scheduled on the node. The reservation guarantees that the pod
+triggering the preemption will be scheduled on that node. As a side effect
+the preempted pods that restart can also not “slip back” into the freed up
+space on the node while the scheduler waits for all preempted pods to exit.
+They could be scheduled on a different node but not stop the pod that
+triggered the preemption from being scheduled.
+
+## Preemption Loop Prevention
+
+The rules defined prevent looping and circular preemptions. Testing should
+be added to cover each rule and combinations like below.
+
+ReplicaSets are a good example to look at for looping and circular
+preemption. Each time a pod from a replica set is removed the ReplicaSet
+controller will create a new pod to make sure the set is complete. That
+auto-recreate could trigger loops if the rules are not covering all cases
+correctly. The case described below should be covered as part of standard
+testing.
+
+Example setup: Replica set _Prod Repl_ runs in queue _Prod_. For testing
+a replica set _Test Repl_ runs in the queue _Test_. Both queues belong
+to the same parent queue (they are siblings). The pods all run with the
+same settings for priority and preemption. There is no space left on the
+cluster.
+
+Example 1: _Prod_ is under its guaranteed quota and multiple pods of the
+replica set are pending. All pods for the _Test Repl_ set are running and
+_Test_ is above its guaranteed resource capacity.
+
+Preemption flow case 1: To make room for a _Prod Repl_ pod, a pod from
+the _Test Repl_ set is preempted. Both queues will end up above their
+guaranteed resource. The _Test Repl_ pod is recreated. It cannot preempt
+the _Prod Repl_ pod as that would leave the _Prod_ queue below its
+guaranteed resource capacity. The _Test Repl_ pod will have to wait until
+resources become available in the cluster. The other pods for the
+_Prod Repl_ set will also have to wait. The _Prod_ queue is now above its
+guaranteed resource and cannot trigger preemption anymore.
+
+Preemption flow case 2: _Test_, although above its guaranteed resource
+capacity, would end up below its guarantee if a pod from _Test Repl_ would be
+preempted. No preemption happens and the _Prod Repl_ pods will stay pending
+until space becomes available in the cluster.
+
+Preemption flow case 3: To make room for the _Prod Repl_ pod a pod from the
+_Test Repl_ set is preempted. The _Prod_ queue, even with the additional pod,
+is still below its guaranteed resource. The _Test Repl_ pod is recreated.
+_Prod_ is not considered as a queue with victims as it is still below its
+guaranteed resources. The _Test Repl_ pod will have to wait until resources
+become available in the cluster. The _Prod_ queue can trigger further
+preemptions. Preemption checks start from scratch. Depending on the state this
+could trigger none or more preemptions from the Test queue.
+
+## Preemption Configuration
+
+### PriorityClass
+
+As part of the pod specification we need to be able to provide the ability to
+opt-out of preemption. Allowing a user to specify the opt-out directly on the
+pod independent of the priority causes a disconnect between Kubernetes and
+YuniKorn preemption configuration. It also removes control from the
+administrator of the cluster to define which priorities should not be able
+to opt out.
+
+One of the goals of the design is to not introduce new objects in Kubernetes.
+Combined with the above control and management of the opt-out functionality
+at a cluster level the design reuses the existing PriorityClass object.
+We cannot and must not break existing functionality of the default Kubernetes
+scheduler when we do this. The plugin version of Kubernetes relies on the
+default scheduler and its behavior. Pods will use the priorityClassName just
+like standard Kubernetes.
+
+The PriorityClass object allows specifying a _preemptionPolicy_ with two
+possible values:
+* PreemptLowerPriority
+* Never
+
+The values are hard-coded in the core types and validations. Adding a new
+value is not possible without making a Kubernetes change. That means we cannot
+reuse the _preemptionPolicy_ field itself. The content of the field will be
+propagated to the scheduling core to allow YuniKorn to take the setting into
+account while scheduling.
+
+Just like all other objects in Kubernetes, the PriorityClass object extends
+the base object and includes metadata. As part of the metadata we can set
+labels and or annotations on the object. Labels are more restrictive than
+annotations. As a generic rule we prefer to use annotations over labels for
+all the extensions we are adding.
+
+The proposal is to add an annotation to the PriorityClass that allows an
+opt-out of preemption. The name of the annotation would follow the same
+structure as we use for all other annotations:
+ `yunikorn.apache.org/allow-preemption`
+
+As the value of the annotation the proposal is to use a boolean value:
+* "true": allow the pod to be preempted,
+* "false": do not allow preemption
+If no value is set for _allow-preemption_ the default value _true_ is used.
+
+The design comes with a caveat. The value of "_false_" for the annotation
+on the PriorityClass provides the possibility for a pod to request not to
+be preempted. However, this is not a guarantee. There could be
+circumstances that the scheduler might still preempt the pod. This will be
+a last resort action if no other solution is available. This means that the
+preemption policy as defined above is a strong suggestion, not a guarantee.
+
+The example use case that could break the rule and preempt a pod with
+_allow-preemption_ set to _false_ is the DaemonSet case. The DaemonSet pod
+must run on the specified node. If all pods that are running on that node
+are either other DaemonSet pods or pods that have _allow-preemption_ set
+to _false_ there is no other choice. A pod that opted out of preemption
+will be preempted.
+
+### Application Settings
+
+Priority is part of the pod specification on Kubernetes. Every pod is
+converted into an AllocationAsk by the k8shim. As part of the DaemonSet
+preemption that was implemented in
+[YUNIKORN-1085](https://issues.apache.org/jira/browse/YUNIKORN-1085) the
+priority of a pod is read and set on the AllocationAsk that is created. The
+priority that is passed on currently is just a plain integer value, we do not
+need more than that.
+
+As part of preemption we need two additional values to be passed on. These two
+values together form the preemptionPolicy for YuniKorn. The values must be
+communicated as part of the AllocationAsk sent from the Kubernetes shim to the
+core. Both values are mapped to the content of the PriorityClass described above.
+Instead of adding two separate fields, wrapping them into a new message will
+provide more clarity.
+
+The preemption policy can be processed for any AllocationAsk, even if it has
+the `Originator` flag in the message set to _false_. Specifying different
+preemption behavior for different asks within an application could be important
+to allow low-cost asks to be preempted while high-cost asks opt-out.
+
+The use case is again a Spark application. You want the driver pod to opt-out
+from preemption and, possibly, preempt other pods during scheduling. An executor
+should do neither.
+
+In protobuf message form that would look like:
+
+```
+message AllocationAsk {
+ ...
+ // The preemption policy for this ask
+ PreemptionPolicy preemptionPolicy = 12;
+}
+
+message PreemptionPolicy {
+ // Allow preemption while running
+ // Based on the YuniKorn annotation from the PriorityClass object
+ bool allowPreemption = 1;
+ // Preemption behavior while scheduling
+ // Based on the K8s preemptionPolicy from the PriorityClass object
+ string schedulerPreemption = 2;
+}
+```
+
+### Queue Configuration
+
+Two new attributes will be added to the queue configuration object to allow
+specifying the preemption fence type and preemption delay. Both properties,
+`preemption.policy` and `preemption.delay`, can be set on any queue. The
+attributes are part of the queue properties as follows:
+
+```yaml
+queues:
+ - name: fencedqueue
+ properties:
+ preemption.policy: fence
+ preemption.delay: 30s
+```
+
+Properties are fully supported in child templates. Support for setting these
+properties for dynamic queues is included as part of the existing
+functionality.
+
+The _preemption.policy_ property can be set on the root queue but has no effect.
+The root queue as the top of the hierarchy has no parent. This means preemption
+cannot traverse further up the tree. No messages will be logged if the
+_preemption.policy_ is set on the root queue.
+
+The supported values for the _preemption.policy_ are:
+* default
+* fence
+* disabled
+
+If the value `fence` is set on a queue, tasks running in or below the queue on
+which the property is set cannot preempt tasks outside the queue tree. The
+value `default` does not limit preemption to the subtree. If no policy is set
+the value `default` is used.
+
+A _preemption.delay_ can only be set on a leaf queue. The leaf queue contains
+the tasks. Tasks trigger the preemption in combination with a queue being below
+its guaranteed resource capacity. Setting a preemption delay on a queue that is
+not a leaf has no effect. No messages will be logged if a _preemption.delay_ is
+set on a parent queue.
+
+The _preemption.delay_ value must be a valid Golang duration in string form. The
+value will be converted into a duration using `ParseDuration` as defined in the
+time package.
+
+A _preemption.delay_, if specified, must be larger than `0s`. If no
+_preemption.delay_ property is specified in the queue the default of `30s` is
+used. If parsing of the string fails the default of `30s` is used.
+
+## Scheduler storage object changes
+
+### AllocationAsk
+
+In line with the changes for the communication the objects in the scheduler also
+need to be modified to persist some of the detail communicated. For the
+AllocationAsk we need to store both new fields.
+
+Proposed new fields: _allowPreemption_ and _schedulerPreemption_
+
+ - _schedulerPreemption_ is only used during the scheduling phase.
+ - _allowPreemption_ will be propagated to the resulting allocation as defined
+ below.
+
+### Allocation
+
+The allocation that gets created based on the AllocationAsk needs a new field.
+This removes the need to check the underlying AllocationAsk if it allows
+preemption or not. The scheduler preemption value is not required in the
+Allocation.
+
+Proposed new field: _allowPreemption_
+
+### Queue
+
+The new preemption policy and delay values will need to be stored in the Queue
+object. The configuration object does not change as we use the properties of
+the Queue in the configuration to store them.
+
+Proposed new fields: _preemptionPolicy_ and _preemptionDelay_
diff --git a/versioned_docs/version-1.5.1/design/priority_scheduling.md b/versioned_docs/version-1.5.1/design/priority_scheduling.md
new file mode 100644
index 0000000000..84f9ad734d
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/priority_scheduling.md
@@ -0,0 +1,408 @@
+---
+id: priority_scheduling
+title: Priority Scheduling
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Introduction
+
+Currently, YuniKorn can sort at two levels: the applications in a queue and
+the resource requests within the application. The queue sorting policy is
+configurable and supports a _Fair_ or _FIFO_ policy. _StateAware_ is
+considered a filtered _FIFO_ policy. Applications cannot be sorted based
+on priority.
+
+Within an application, requests are sorted based on priority of the requests.
+For requests that have the same priority the submission time is used as a
+secondary sort key. Requests are sorted in _FIFO_ order if they have the
+same priority. Note that the priority of requests was not propagated from
+the shim to the core until
+[YUNIKORN-1235](https://issues.apache.org/jira/browse/YUNIKORN-1235) was
+implemented in v1.0.0. Until that implementation all requests were created
+with the _default_ priority.
+
+Additionally, Kubernetes uses priority as a sort when performing preemption.
+YuniKorn currently only does preemption of requests if a node-specific task
+needs to be scheduled (i.e. a DaemonSet pod,
+[YUNIKORN-1085](https://issues.apache.org/jira/browse/YUNIKORN-1085)).
+
+## Goals
+
+- Attempt to be compatible with Kubernetes standard priority handling
+ wherever possible
+- Priority should be orthogonal to application sorting policies (i.e.
+ _Fair_, _FIFO_ and _StateAware_ policies should support prioritization)
+- It should be possible to limit the scope of priority handling
+- Design of priority scheduling should consider the impact on preemption
+ design
+
+## Non Goals
+
+- Design of preemption
+- Change AllocationAsk sorting within an application
+
+## Definitions
+
+The following terms are defined as follows and will be used throughout this
+document.
+
+### Priority
+
+Priority is a numeric value associated with an Ask. Higher-priority asks have
+higher numeric values. To be compatible with Kubernetes, all 32-bit signed
+integers are considered valid priorities. An ask without a specific defined
+priority is assumed to have the default priority.
+
+### Minimum Priority
+
+Minimum priority is defined as the lowest possible priority, or
+`-2,147,483,648` (- 2<sup>31</sup>).
+
+### Maximum Priority
+
+Maximum priority is defined as the highest possible priority, or
+`2,147,483,647` (2<sup>31</sup> - 1). Note that Kubernetes does not permit
+custom priority classes (see below) to define priorities greater than
+`1,000,000,000` (1 billion). Values higher are reserved for system-level
+priorities.
+
+### Default Priority
+
+Default priority is defined as the value `0`.
+
+### PriorityClass
+
+A [PriorityClass](https://kubernetes.io/docs/concepts/scheduling-eviction/pod-priority-preemption/#priorityclass)
+is a Kubernetes object which maps between a priority name and an integer
+priority value. The name cannot start with `system-` as that prefix is
+reserved for Kubernetes internal usage.
+
+Additionally, a `preemptionPolicy` may be set on a PriorityClass. Official
+GA support was added in Kubernetes 1.24. The preemption policy is only
+used while scheduling the pod itself. It is not taken into account after
+the pod is scheduled.
+
+The policy defaults to `Never`, implying that pods of this priority will
+never preempt other pods. `PreemptLowerPriority` is also possible, which
+allows the scheduler to preempt pods with lower priority values.
+
+When a pod is submitted with its `priorityClassName` set to a valid priority
+class, a Kubernetes-provided admission controller will automatically update
+the `priority` and `preemptionPolicy` fields of the pod to match those of the
+referenced PriorityClass. Pods that specify a priority as part of the
+specification are rejected if the integer value set on the pod does not match
+the PriorityClass value.
+
+#### Pre-defined PriorityClass objects
+- system-cluster-critical: `2,000,000,000`
+- system-node-critical: `2,000,000,100`
+
+## Current state
+
+The framework for priority is minimally implemented within YuniKorn. The
+_AllocationAsk_ structure within the scheduler interface already contains a
+_priority_ field which is populated by the shim based on the _priority_ field
+on a Pod.
+
+One priority can be set on creation of the _AllocationAsk_. Each
+_AllocationAsk_ within an application may have its own priority.
+_AllocationAsk_ sorting is based on this priority. This means that within the
+application asks or pods are sorted based on priority. High-priority asks will
+be scheduled before low-priority asks.
+
+Priorities are currently also used for the minimal DaemonSet preemption process
+implemented in [YUNIKORN-1085](https://issues.apache.org/jira/browse/YUNIKORN-1085).
+Possible victims are sorted based on their priority among other things. The
+priority of the _Allocation_ is the same as the _AllocationAsk_ they
+represent.
+
+One use of the capability to set a priority for each _AllocationAsk_ could be to
+enable dynamic allocations such as those utilized by Apache Spark. An
+application can submit a set of asks at high priority that will be scheduled
+as soon as possible, and a set of lower-priority asks that can be used for
+additional capacity if available.
+
+## Proposal
+
+### Priority fencing
+
+By default, priority applies across queues globally. In other words, a
+higher-priority ask will be attempted to be satisfied before a lower-priority
+ask regardless of which queue the asks reside in.
+
+However, it is often useful to limit the scope of priority handling. For
+example, it may be desirable to limit the ability of asks in one area of the
+queue hierarchy from jumping ahead of asks in another area. To support this,
+we add the concept of a _Priority Fence_. A queue (either _leaf_ or _parent_)
+may be configured as priority-fenced, which prevents priorities from
+propagating upward.
+
+If a queue is fenced, asks within that queue (and subqueues) will be
+prioritized relative to each other, but not relative to asks in other portions
+of the queue hierarchy. A priority fence is a YuniKorn queue subtree boundary
+above which priorities are no longer considered. A priority fenced queue does
+not expose priority levels of its children to its parent queue, thereby
+ensuring that priorities are only considered within that queue and its
+descendants.
+
+Priority fences may be nested; for example a fenced queue `tenant 1` may have
+child queues `queue A` (fenced) and `queue B` (unfenced). In this case tasks
+in `queue A` will show as the default priority and will not be prioritized
+above `queue B`, but tasks in `queue B` may be prioritized over those in
+`queue A`. In no case will tasks in either queue be prioritized over tasks
+outside the scope of queue `tenant 1`, as they show as the default priority
+(shown in the diagram below with yellow arrows).
+
+A similar setup is created for the `tenant 2` queue. However in this case the
+priorities from `queue 1` to `queue 2` and vice-versa are fully visible
+(shown in the diagram below with blue arrows).
+
+None of the tasks running in the queues below `tenant 1` or `tenant 2` can be
+prioritized above the `system` queue. Tasks in the `system` queue are not
+fenced. That means that a task in the `system` queue can be prioritized above
+any queue in the hierarchy (shown in the diagram below with red arrows).
+
+
+
+### Priority offset
+
+Priorities need to be pre-defined at the cluster level via the PriorityClass
+objects. At the Kubernetes level there is no way to limit which priorities can
+be used in which part of the system. The only exception to that rule are the
+pre-defined system priorities from the Kubernetes system itself. This does not
+give an administrator a lot of control.
+
+Prioritizing a specific queue above another queue based on the priorities of
+the tasks inside the queue is thus difficult. Any user can start a task with
+any priority anywhere in the cluster. To allow an administrator more control
+we need to be able to steer, or augment, the priority of a queue. To
+accomplish this, we introduce the `priority.offset` property of a queue.
+
+The `priority.offset` property on a queue allows the administrator to increase
+(or decrease) a queue's priority. By using the offset a queue can be boosted
+to become a high-priority queue. It also can be used to decrease the priority
+compared to its siblings.
+
+As a general rule: the priority of a _leaf_ queue is equal to the maximum
+priority of the AllocationAsks in that queue, plus the `priority.offset`
+value. The priority of a _parent_ queue is the maximum of the priority of its
+child queues plus the `priority.offset` value. This means that priority
+offsets are additive when traversing up the queue hierarchy. A priority offset
+may be any valid signed `int32` value. The offset is applied after the dynamic
+priority for the queue is calculated.
+
+As a note of caution: a side effect of a very large offset value for a queue
+could result in all pods in that queue being clamped at the maximum priority
+level or exceeding the priority of Kubernetes system priority classes. This
+could then adversely impact the scheduling of node and or cluster critical
+pods in sibling queues. This effect needs to be clearly documented. Logging a
+warning about a configured value that high should also be considered.
+
+A second option that was considered is limiting the offset value to
+`999,999,999`. This would prevent the offset priority from exceeding the
+pre-defined system priorities. This would be a safe option but does not take
+into account the fact that we can have negative priorities that we want to
+offset. The current choice is to go with the first option to document and log.
+
+Fencing affects the priority calculation for the queue. When a queue is
+fenced (i.e. the `priority.policy` is set to `fence`), queue priority
+calculations are disabled for that queue. The result of application priority
+or child queue calculations are ignored for a fenced queue. The priority of
+the fenced queue is always set to the default priority or to the priority
+offset, if specified.
+
+### Extended application sorting
+
+Applications are currently sorted based on one of the defined application
+sorting policies. All application sorting policies only take into account
+applications that have pending requests.
+
+Currently defined are _Fair_, _FIFO_, and _StateAware_:
+
+- _Fair_ sorting sorts based on relative usage within a queue
+- _FIFO_ sorts applications by creation time in a first-in, first-out manner
+- _StateAware_ is similar to _FIFO_ except that only a single "new"
+ application is allowed at once (this is an over-simplification, but will
+ suffice here)
+
+To support priority scheduling, the policies _Fair_ and _FIFO_ will get an
+equivalent policy that considers the priority of an application first. For
+applications that have the same priority, sorting will then fall back to the
+secondary sort criteria (_Fair_ or _FIFO_).
+
+_StateAware_ as a special form of _FIFO_ will however always filter
+applications first and then sort. A priority based version of _StateAware_
+must change the way it filters. The single "new" application that is
+considered can no longer be the oldest application. That would be a major
+behavioral change from the current policy.
+
+The existing policies will not be changed to maintain backwards
+compatibility. None of the existing policies have a tiebreak built in. In
+the case the comparison returns equality the order of the elements is not
+changed.
+
+Instead of introducing new policies to add priority to the existing
+behavior a flag will be added that allows switching priorities on or off.
+The property will be added to the queue.
+
+New property: `appication.sort.priority`
+
+### Extended queue sorting
+
+Queue sorting currently only supports one policy: _Fair_. Queues cannot
+be sorted using a _FIFO_ mechanism. _FIFO_ sorting of queues is not in
+scope of this design. However the design has allowed for a simple addition
+of a new policy.
+
+The _Fair_ policy works similarly to application _Fair_ sorting, except
+that it sorts based on usage ratios between two queues. Allocations are
+compared to guaranteed resources. This results in the queue furthest below
+its guaranteed resource to be first in the sorted queue list as it is
+considered "more starved". For queues that have the same usage ratio,
+the queue with the highest pending requests is considered more starved.
+
+To add priority support, a new property will be added to consider the
+current queue priority as the main sorting factor. To break the tie between
+queues with the same priority, starvation like in the _Fair_ policy will be
+used. Again breaking the tie for queues with the same _Fair_ usage using the
+pending requests.
+
+The existing policy will not be changed to maintain backwards compatibility.
+The same property as used for application sorting will be reused to control
+priority behavior.
+
+New property: `application.sort.priority`
+
+### Priority Configuration
+
+#### Queue Configuration
+
+Three new attributes will be added to the queue configuration object to allow
+specifying the priority fence type and priority offset and priority sorting.
+All properties `priority.policy`, `priority.offset`, and
+`application.sort.priority`, can be set on any queue. The attributes are part
+of the queue properties as follows:
+
+```yaml
+queues:
+ - name: fencedqueue
+ properties:
+ priority.policy: fence
+ priority.offset: 100
+ application.sort.priority: enabled
+```
+
+The `priority.policy` and `priority.offset` fields are not inherited from the
+parent queue to a child queue; they apply only to the queue they are specified
+on.
+
+Note that the `application.sort.priority` field is only inherited if the value
+is set to `disabled` (the `enabled` value is already the default). This allows
+priorities to be ignored for an entire queue subtree by setting the property
+on the parent queue.
+
+Properties are fully supported in child templates. Support for setting these
+properties for dynamic queues is included as part of the existing
+functionality.
+
+The supported, not case sensitive, values for the `application.sort.priority`
+are:
+
+ - `enabled`
+ - `disabled`
+
+The proposal is to default to `enabled` for the `application.sort.priority`
+property. Given that the Kubernetes default scheduler and preemption support
+are all based on priorities that choice seems more logical than `disabled`.
+
+The `application.sort.priority` when set on a _parent_ queue affects the way
+the queues are sorted. It will change queue sorting from the standard _Fair_
+sorting to priority-based sorting. To break the tie for equal priority
+queues it will use the _Fair_ comparison. When set on a _leaf_ queue it will
+change the application sorting itself. All three existing policies will be
+supported with `application.sort.priority` set to `enabled` or `disabled`.
+For the `disabled` case nothing will change and the old behavior is
+maintained. For the `enabled` case the primary sort will be priority and the
+configured policy (_Fair_, _FIFO_, or _StateAware_) will be used as
+tiebreakers.
+
+The `priority.policy` and `priority.offset` can be set on the `root` queue
+but have no effect. The root `queue` as the top of the hierarchy has no
+parent. This means priority cannot traverse further up the tree. No messages
+will be logged if the `priority.policy` or `priority.offset` are set on the
+`root` queue.
+
+The supported, not case sensitive, values for the `priority.policy` are:
+
+- `default`
+- `fence`
+
+If the value `fence` is set on a queue, the queue on which the property is
+set does not propagate its priority outside of the queue's subtree. The value
+`default` does not fence the queue. If no policy is set the value `default` is
+used.
+
+The `priority.offset` value must be a valid integer in string form. The value
+will be converted into an integer using the standard parsing methods as defined
+via: [`strconv.ParseInt`](https://pkg.go.dev/strconv#ParseInt)`(s, 10, 32)`.
+
+If no `priority.offset` property is specified in the queue the default of `0`
+is used. Specifying the property with an empty value is equivalent to `0`. If
+parsing of the string fails the default of `0` is used.
+
+### Scheduler storage object changes
+
+#### Application
+
+The _Application_ object within the scheduler core will be updated with a
+dynamic priority value, defined as the maximum priority of all pending asks.
+To avoid sorting performance degradation, this value should be cached and
+recomputed only when allocation asks are added, allocated, or removed.
+
+This recalculation can be avoided in many instances. When new asks are added,
+an application's priority need not be recalculated unless the priority of the
+new ask is greater than that of the application as a whole. Similarly, when
+removing an ask, the priority need not be adjusted unless the ask's priority
+is equal to the application's priority. When recalculating priority due to
+adding a high-priority ask, the application's priority can simply be set to
+the new ask's priority. When recalculating due to removal of an ask of equal
+priority to the application, we can stop considering additional asks as soon
+as any other ask of equal priority is encountered. This means that the
+performance impact of application priority updates will be `O(1)` when adding
+an ask, and at worst `O(n)` when removing an ask.
+
+New field: `askMaxPriority`
+
+#### Queue
+
+The new configuration values need to be maintained on the queue. Besides those
+values the _Queue_ object within the scheduler core will be updated with a
+dynamic priority value. This dynamic priority will be defined as the maximum
+priority of all applications within the queue, plus any "priority offset"
+configured.
+
+To avoid sorting performance degradation, this value should be cached and
+recomputed only when necessary. For a _leaf_ queue, this is required if
+application priorities in the queue have changed, and for a _parent_ queue this
+is required if the priorities of child queues have changed.
+
+New fields: `priorityPolicy`, `priorityOffset`, `currentPriority`
+
diff --git a/versioned_docs/version-1.5.1/design/resilience.md b/versioned_docs/version-1.5.1/design/resilience.md
new file mode 100644
index 0000000000..ac0c93bac9
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/resilience.md
@@ -0,0 +1,144 @@
+---
+id: resilience
+title: Resilience
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+This is not a HA (High-availability) design, HA implies that a service can
+survive from a fatal software/hardware failure. That requires one or more
+standby instances providing same services to take over active instance on failures.
+Resilience here means for YuniKorn, we can restart it without losing its state.
+
+## The problem
+
+YuniKorn is designed as a stateless service, it doesn't persist its state, e.g
+applications/queues/allocations etc, to any persistent storage. All states are
+in memory only. This design ensures YuniKorn to be able to response requests with
+low latency, and deployment mode is simple. However, a restart (or recovery) will
+have the problem to lose state data. We need a decent way to reconstruct all
+previous states on a restart.
+
+## Design
+
+### Workflow
+
+Scheduler core has no notion of "state", which means it does not know if it is under recovering.
+It is too complex to maintain a series of `scheduler states` in both core and shim, because we must
+keep them in-sync. However, if we live under a simple assumption: **scheduler core only responses
+requests, the correction of requests is ensured by shim according its current state**.
+The design becomes much simpler. This way, the shim maintains a state machine like below. When
+it is under `running` state, it sends new requests to the scheduler core as long as a new one is found;
+when under `recovering` state, it collects previous allocations and send recovery messages to
+the scheduler core, and waiting for recovery to be accomplished.
+
+Shim scheduler state machine
+
+```
+ Register Recover Success
+New -----------> Registered -----------> Recovering ----------> Running
+ | Fail
+ --------> Failed
+```
+
+Following chart illustrate how yunikorn-core and shim works together on recovery.
+
+
+
+Restart (with recovery) process
+- yunikorn-shim registers itself with yunikorn-core
+- shim enters "recovering" state. Under "recovering" state, the shim only scans existing nodes and allocations, no new scheduling requests will be sent.
+ - shim scans existing nodes from api-server and added them to cache
+ - shim scans existing pods from api-server, filter out the pods that already assigned (scheduled to a node), and added that to cache (allocation in that node)
+ - shim sends update request to yunikorn-core with the info found in previous steps
+- yunikorn-core handles update requests, the steps should look like a replay of allocation process, including
+ - adding node
+ - adding applications
+ - adding allocations
+ - modifying queue resources
+ - update partition info
+- when all nodes are fully recovered, shim transits the state to "running"
+- shim notifies yunikorn-core that recovery is done, then yunikorn-core transits to "running" state.
+
+### How to determine recovery is complete?
+
+Shim queries K8s api-server to get how many nodes were available in this cluster. It tracks the recovering status of each node.
+Once all nodes are recovered, it can claim the recovery is completed. This approach requires us to add `recovering` and `recovered`
+states to nodes' state machine in the shim.
+
+### Node recovery
+
+In the shim layer, it maintains states for each node and pods running on this node. When start to recover nodes,
+all nodes initially are considered as under `recovering`. Only when all pods running on this node are fully recovered,
+the node can be considered as `recovered`.
+
+
+
+Like demonstrated on upon diagram,
+
+- Node0 is still recovering because pod0 is recovering.
+- Node1 is recovered (become schedulable) because all pods on this node have been recovered.
+- Node2 is lost, shim lost contact with this node. If after sometime this node comes back, shim should still try to recover this node.
+
+### Requests for recovery
+
+During recovery process, shim needs to collect all known information of applications, nodes and allocations from the underneath
+Resource Manager and use them for recovery.
+
+#### Applications
+
+Existing applications must be recovered first before allocations. Shim needs to scan all existing applications
+from nodes, and add applications info as a list of `AddApplicationRequest` in the `UpdateRequest`. This is same
+as the fresh application submission.
+
+```
+message AddApplicationRequest {
+ string applicationID = 1;
+ string queueName = 2;
+ string partitionName = 3;
+}
+```
+
+#### Nodes and allocations
+
+Once a shim is registered to the scheduler-core, subsequent requests are sent via `UpdateRequest#NewNodeInfo`
+(see more from [si.proto](https://github.com/apache/yunikorn-scheduler-interface/blob/master/si.proto)).
+The structure of the messages looks like,
+
+```
+message NewNodeInfo {
+ // nodeID
+ string nodeID = 1;
+ // optional node attributes
+ map<string, string> attributes = 2;
+ // total node resource
+ Resource schedulableResource = 3;
+ // existing allocations on this node
+ repeated Allocation existingAllocations = 4;
+}
+```
+Shim needs to scan all existing allocations on a node and wrap these info up as a `NewNodeInfo`, add that to a
+`UpdateRequest` and then send to scheduler-core.
+
+**Note**: the recovery of existing allocations depend on the existence of applications, which means applications must
+be recovered first. Since scheduler-core handles `UpdateRequest` one by one, it is required that all existing allocations
+in a `UpdateRequest` must from known applications or new applications embedded within the same `UpdateRequest`, which can be
+specified in `NewApplications` field. Scheduler-core ensures `NewApplications` are always processed first.
+
diff --git a/versioned_docs/version-1.5.1/design/scheduler_configuration.md b/versioned_docs/version-1.5.1/design/scheduler_configuration.md
new file mode 100644
index 0000000000..54b616a4ec
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/scheduler_configuration.md
@@ -0,0 +1,246 @@
+---
+id: scheduler_configuration
+title: Scheduler Configuration
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The Yunikorn core scheduler configuration has two separate areas that need to be configured. The scheduler service itself, things like web service ports etc, and the queue configuration. The split between the two types of configuration is proposed with two points in mind:
+* Separation of duty
+* Dynamic vs Static
+
+The scheduler configuration is mainly static. There is no need to change a web service port or a scheduling policy while the service is running. The queue configuration is far more dynamic and can change while the service is running.
+
+From a separation of duty we can allow an operator that manages the cluster to make changes to the scheduler queues. You would not want to allow that administrator to change the scheduler configuration itself.
+
+Separated from the core scheduler configuration we have one or more shim configurations. We currently cannot anticipate the deployment model of the scheduler and its shims. A shim, like the k8s-shim, might run in the same container or node but there is no guarantee it will. We also do not know the number of shims that will be used with one core scheduler. There is also still the possibility to have multiple instances of the same shim with one core scheduler.
+
+Shim configuration must be independent of the core scheduler configuration.
+## Scheduler Configuration
+Scheduler configuration covers all the configuration needed to start the scheduler and the dependent services. The configuration consists of a simple key value pair. All configuration to start the service must be part of this configuration.
+The scheduler configuration must exclude the queue related configuration.
+
+Scheduler configuration as currently identified
+* Bind host
+* Service port
+* Web bind host
+* Web service port
+* SSL config
+* Shims Configured
+* SchedulerACL
+
+Configuration to consider:
+* Assign multiple containers in one go: use case is bin packing, don’t spread an application over large number of nodes. Needs to become configurable.
+* Pre-emption related configuration:
+ * threshold: do not pre-empt from a queue if the cluster load is below a certain threshold.
+ * Interval: pause between pre-emption checks
+## Queue Configuration
+### Queue Definition
+On startup the scheduler will load the configuration for the queues from the provided configuration file after initialising the service. If there is no queue configuration provided the scheduler should start up with a simple default configuration which performs a well documented default behaviour.
+Based on the kubernetes definition this configuration could be a configMap <sup id="s1">[1](#f1)</sup> but not a CRD.
+
+The queue configuration is dynamic. Changing the queue configuration must not require a scheduler restart.
+Changes should be allowed by either calling the GO based API, the REST based API or by updating the configuration file. Changes made through the API must be persisted in the configuration file. Making changes through an API is not a high priority requirement and could be postponed to a later release.
+
+The queue configuration defines queues in a hierarchy: a tree. The base of the tree is the _root_ queue. The queue configuration must define a single _root_ queue. All queues that are defined in queue configuration are considered _managed_ queues.
+
+The root queue reflect the whole cluster. Resource settings on the root queue are not allowed. The resources available to the root queue are calculated based on the registered node resources in the cluster. If resources would be specified on the root limit the cluster would either be artificially limited to a specific size or expect resources to be available that are not there.
+
+Queues in the hierarchy in the tree are separated by the “.” dot character (ASCII 0x2E). This indirectly means that a queue name itself cannot contain a dot as it interferes with the hierarchy separator. Any queue name in the configuration that contains a dot will cause the configuration to be considered invalid. However we must allow placement rules to create a queue with a dot based input.
+
+Not all queues can be used to submit an application to. Applications can only be submitted to a queue which does not have a queue below it. These queues are defined as the _leaf_ queues of the tree. Queues that are not a _leaf_ and thus can contain other queues or child queues are considered _parent_ queues.
+
+Each queue must have exactly one _parent_ queue, besides the root queue. The root queue cannot have a _parent_ and will be automatically defined as a _parent_ queue type.
+A fully qualified queue name, case insensitive, must be unique in the hierarchy. A queue in the hierarchy can thus be only uniquely identified by its fully qualified path. This means that a queue with the same name is allowed at a different point in the hierarchy.
+Example:
+```
+root.companyA.development
+root.companyB.development
+root.production.companyA
+```
+In the example the queues _companyA_ and _companyB_ are _parent_ queues. Both _development_ queues are _leaf_ queues.
+The second instance of the _companyA_ queue is a _leaf_ queue which is not related to the first instance as it is defined at a different level in the hierarchy.
+
+The queue as defined in the configuration will be assigned a queue type. This can either be implicit based on how the queue is defined in the hierarchy or explicit by setting the optional _parent_ property as part of the queue definition. By default all queues will be assigned their type based on the configuration. There is only one case in which this should automatic process would need to be overridden and that is to mark a _leaf_ in the configuration as a _parent_. The use case is part [...]
+
+Access control lists provide a split between submission permission and administration permissions. Submission access to a queue allows an application to be submitted to the queue by the users or groups specified. The administration permissions allows submission to the queue plus the administrative actions. Administrative actions are currently limited to killing an application and moving an application to a different queue.
+
+Access control lists are checked recursively up to the root of the tree starting at the lowest point in the tree. In other words when the access control list of a queue does not allow access the parent queue is checked. The checks are repeated all the way up to the root of the queues.
+
+On each queue, except the root queue, the following properties can be set:
+* QueueType:
+ * Parent (boolean)
+* Resource settings:
+ * Guaranteed (resource)
+ * Maximum (resource)
+* Running Application limit:
+ * Maximum (integer)
+* Queue Permissions:
+ * SubmitACL (ACL)
+ * AdminACL (ACL)
+* Pre emption setting:
+ * PreEmptionAllowed (boolean)
+* Application sort algorithm:
+ * ApplicationSortPolicy (enumeration: fair, fifo)
+
+On the root queue only the following properties can be set:
+* Running Application limit:
+ * Maximum (integer)
+* Queue Permissions:
+ * SubmitACL (ACL)
+ * AdminACL (ACL)
+* Application sort algorithm:
+ * ApplicationSortPolicy (enumeration: fair, fifo)
+
+### User definition
+Applications are run by a user could run in one or more queues. The queues can have limits set on the resources that can be used. This does not limit the amount of resources that can be used by the user in the cluster.
+
+From an administrative perspective setting a limit of the resources that can be used by a specific user can be important. In this case a user is broadly defined as the identity that submits the application. This can be a service or a person, from a scheduling perspective there is no difference.
+User limits can prevent a take over of a queue or the cluster by a misbehaving user or application. From a multi tenancy perspective user limits also allows for sharing or subdivision of resources within the tenancy however that is defined.
+
+Adding user based limits will allow the cluster administrators to control the cluster wide resource usage of a user:
+* Running Application limit:
+ * Maximum (integer)
+* Resource setting:
+ * Maximum (resource)
+
+### Placement Rules definition
+Schedulers can place an application in a queue dynamically. This means that an application when submitted does not have to include a queue to run in.
+
+A placement rule will use the application details to place the application in the queue. The outcome of running a placement rule will be a fully qualified queue or a `fail`, which means execute the next rule in the list. Rules will be executed in the order that they are defined.
+
+During the evaluation of the rule the result could be a queue name that contains a dot. This is especially true for user and group names which are POSIX compliant. When a rule generates a partial queue name that contains a dot it must be replaced as it is the separator in the hierarchy. The replacement text will be `_dot_`
+
+The first rule that matches, i.e. returns a fully qualified queue name, will halt the execution of the rules. If the application is not placed at the end of the list of rules the application will be rejected. Rules can return queues that are not defined in the configuration only if the rule allows creation of queues.
+
+These queues created by the placement rules are considered _unmanaged_ queues as they are not managed by the administrator in the configuration. An administrator cannot influence the _unmanaged_ queue creation or deletion. The scheduler creates the queue when it is needed and removes the queue automatically when it is no longer used.
+
+Rules provide a fully qualified queue name as the result. To allow for deeper nesting of queues the parent of the queue can be set as part of the rule evaluation. The rule definition should allow a fixed configured fully qualified parent to be specified or it can call a second rule to generate the parent queue. By default a queue is generated as a child of the root queue.
+
+Example:
+Placing an application submitted by the user _user1_ whom is a member of the groups _user1_ and _companyA_ in a queue based on UserName:
+```
+Rule name: UserName
+ Parent: root.fixedparent
+Result: root.fixedparent.user1
+
+Rule name: UserName
+ Parent: SecondaryGroup
+ Filter:
+ Type: allow
+ Groups: company.*
+Result: root.companyA.user1
+
+Rule name: UserName
+Filter:
+ Users: user2,user3
+Result: denied placement
+```
+The default behaviour for placing an application in a queue, which would do the same as using the queue that is provided during submit, would be a rule that takes the provided queue with the create flag set to false.
+
+Access permissions will be enforced as part of the rule evaluation. For _managed_ queues this means that the ACL for the queue itself is checked. For an _unmanaged_ queue the parent queue ACL is the one that is checked. For the definition of the access control list and checks see the [Access Control Lists](#access-control-lists) chapter.
+
+Defining placement rules in the configuration requires the following information per rule:
+* Name:
+ * Name (string)
+* Parent
+ * Parent (string)
+* Create Flag:
+ * Create (boolean)
+* Filter:
+ * A regular expression or list of users/groups to apply the rule to.
+
+The filter can be used to allow the rule to be used (default behaviour) or deny the rule to be used. User or groups matching the filter will be either allowed or denied.
+The filter is defined as follow:
+* Type:
+ * Type (string) which can have no value (empty) or "allow" or "deny", case insensitive.
+* Users:
+ * A list of zero or more user names. If the list is exactly one long it will be interpreted as a regular expression.
+* Groups:
+ * A list of zero or more group names. If the list is exactly one long it will be interpreted as a regular expression.
+
+Proposed rules for placing applications would be:
+* Provided: returns the queue provided during the submission
+* UserName: returns the user name
+* PrimaryGroupName: returns the primary group of the user
+* SecondaryGroupName: returns the first secondary group of the user that matches
+* Fixed: returns the queue name configured in the rule
+* ApplicationType: returns the application type (if available)
+
+For _unmanaged_ queues in the current revision of the configuration you cannot provide any queue specific properties. However in the future we should consider propagating specific resource related settings from a _managed_ parent to the _unmanaged_ child, specifically:
+* Dynamic Resource settings:
+ * Guaranteed (resource)
+ * Maximum (resource)
+* Dynamic Running Application limit:
+ * Maximum (integer)
+
+### Configuration updates
+Updating the queue definition will allow updating the existing queue properties as well as adding and removing queues. A new queue definition will only become active if the configuration can be parsed. The change of the definition is an atomic change which applies all modification in one action.
+
+Updating the queue properties will not automatically trigger further action. This means that if the maximum number of resources of a queue or its parent is changed we leave the applications in the queue running as they are. The scheduler will adhere to the new property values which should see the convergence over time.
+
+A _managed_ queue will only be removed if it is removed from the configuration. Before we can remove a queue it must not be running applications. This means that when a _managed_ queue is removed from the configuration it must be empty or the system needs to allow the queue to drain. Forcing a _managed_ queue to be empty before we can remove it is not possible which means that _managed_ queues are removed in multiple steps:
+1. The queue is removed from the configuration
+1. The queue is marked as `draining`
+1. All managed queues that are `draining` and empty are removed
+
+Long running applications should be handled gracefully when removing a _managed_ queue. The scheduler should at least track and expose that a queue has been in a _draining_ state for an extended period of time. In the optimal case the application should be notified of the queue change to allow it to release resources. In all cases the queue administrators should be notified to allow them to take action. This action would currently be a manual move of the application to a different queue [...]
+
+_Unmanaged_ queues that are not defined in the queue definition are created by the scheduler automatically based on the placement rules. _Unmanaged_ queues have a lifespan independent of the configuration. Whenever an _unmanaged_ queue is empty it will get removed. The queue will automatically be created again when a new application is requesting it via triggering the placement rule.
+
+Removing an empty _managed_ or _unmanaged_ queue is handled by the same removal code which must run independent of the configuration updates and scheduling actions.
+
+Configurations can change over time. The impact of a fail over or restart must still be investigated.
+Base point to make: a changed configuration should not impact the currently running applications. Queues that no longer exist should be handled somehow.
+
+### Access Control Lists
+The scheduler ACL is independent of the queue ACLs. A scheduler administrator is not by default allowed to submit an application or administer the queues in the system.
+
+All ACL types should use the same definition pattern. We should allow at least POSIX user and group names which uses the portable filename character set <sup id="s2">[2](#f2)</sup>. However we should take into account that we could have domain specifiers based on the environment that the system runs in (@ sign as per HADOOP-12751).
+
+By default access control is enabled and access is denied. The only special case is for the core scheduler which automatically adds the system user, the scheduler process owner, to the scheduler ACL. The scheduler process owner is allowed to make sure that the process owner can use the API to call any administrative actions.
+
+Access control lists give access to the users and groups that have been specified in the list. They do not provide the possibility to explicitly remove or deny access to the users and groups specified in the list.
+
+The access control list is defined as:
+```
+ACL ::= “*” | userlist [ “ “ grouplist ]
+userlist ::= “” | user { “,” user }
+grouplist ::= “” | group { “,” group }
+```
+
+This definition specifies a wildcard of * which results in access for everyone. If the user list is empty and the group list is empty nobody will have access. This deny all ACL has two possible representations:
+* an empty access control list.
+* a single space.
+
+If there is no access control list is configured access is denied by default.
+## Shim Configuration
+The shim configuration is highly dependent on the shim implementation. The k8s shim differs from the YARN shim. Currently the k8s shim is configured via command line options but we should not depend on that.
+
+### K8s shim
+The full configuration of the K8s shim is still under development.
+
+### YARN shim
+The full configuration of the YARN shim is still under development.
+
+---
+<br/><b id="f1"></b>1: https://kubernetes.io/docs/concepts/extend-kubernetes/api-extension/custom-resources/#should-i-use-a-configmap-or-a-custom-resource. [↩](#s1)
+<br/><b id="f2"></b>2: The set of characters from which portable filenames are constructed. [↩](#s2)
+<br/>`A B C D E F G H I J K L M N O P Q R S T U V W X Y Z a b c d e f g h i j k l m n o p q r s t u v w x y z 0 1 2 3 4 5 6 7 8 9 . _ -`
diff --git a/versioned_docs/version-1.5.1/design/scheduler_plugin.md b/versioned_docs/version-1.5.1/design/scheduler_plugin.md
new file mode 100644
index 0000000000..f26b86abc1
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/scheduler_plugin.md
@@ -0,0 +1,112 @@
+---
+id: scheduler_plugin
+title: K8s Scheduler Plugin
+---
+
+<!--
+* Licensed to the Apache Software Foundation (ASF) under one
+* or more contributor license agreements. See the NOTICE file
+* distributed with this work for additional information
+* regarding copyright ownership. The ASF licenses this file
+* to you under the Apache License, Version 2.0 (the
+* "License"); you may not use this file except in compliance
+* with the License. You may obtain a copy of the License at
+*
+* http://www.apache.org/licenses/LICENSE-2.0
+*
+* Unless required by applicable law or agreed to in writing, software
+* distributed under the License is distributed on an "AS IS" BASIS,
+* WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+* See the License for the specific language governing permissions and
+* limitations under the License.
+-->
+
+## Background
+
+YuniKorn (on Kubernetes) is traditionally implemented as a ground-up implementation of a Kubernetes scheduler.
+This has allowed us to innovate rapidly, but is not without its problems; we currently have numerous places
+where we call into non-public K8S source code APIs with varying levels of (code) stability, requiring
+sometimes very disruptive code changes when we switch to new Kubernetes releases.
+
+Ideally, we should be able to take advantage of enhancements to new Kubernetes releases automatically.
+Using the plugin model enables us to enhance the Kubernetes scheduling logic with YuniKorn features.
+This also helps keep YuniKorn compatible with new Kubernetes releases with minimal effort.
+
+Additionally, it is desirable in many cases to allow non-batch workloads to bypass the YuniKorn scheduling
+functionality and use default scheduling logic. However, we have no way to do that today as the default
+scheduling functionality is not present in the YuniKorn scheduler binary.
+
+Since Kubernetes 1.19, the Kubernetes project has created a stable API for the
+[Scheduling Framework](https://kubernetes.io/docs/concepts/scheduling-eviction/scheduling-framework/),
+which allows plugins to be created which implement various extension points. Plugins implement one or more
+of these extension points, and are then compiled into a scheduler binary which contains the default
+scheduler and plugin code, configured to call into the plugins during normal scheduling flow.
+
+## Design
+
+We have added a scheduler plugin to the k8s-shim codebase which can be used to build a Kubernetes
+scheduler binary that includes YuniKorn functionality as well as the default scheduler functionality,
+significantly improving the compatibility of YuniKorn with upstream Kubernetes and allowing deployment of
+YuniKorn as the sole scheduler in a cluster with much greater confidence.
+
+Separate docker images are created for the scheduler. The traditional YuniKorn scheduler is built as
+`scheduler-{version}` while the new plugin version is built as `scheduler-plugin-{version}`. Either can be
+deployed interchangeably into a Kubernetes cluster with the same helm charts by customizing the scheduler
+image to deploy.
+
+## Entrypoints
+
+The existing shim `main()` method has been relocated to `pkg/cmd/shim/main.go`, and a new `main()` method
+under `pkg/cmd/schedulerplugin/main.go` has be created. This method instantiates the default Kubernetes
+scheduler and adds YuniKorn to it as a set of plugins. It also modifies the default scheduler CLI argument
+parsing to add YuniKorn-specific options. When the YuniKorn plugin is created, it will launch an instance
+of the existing shim / core schedulers in the background, sync all informers, and start the normal YuniKorn
+scheduling loop.
+
+## Shim Scheduler Changes
+
+In order to cooperate with the default scheduler, the shim needs to operate slightly differently when in
+plugin mode. These differences include:
+
+ - In `postTaskAllocated()`, we don’t actually bind the Pod or Volumes, as this is the responsibility of
+ the default scheduler framework. Instead, we track the Node that YK allocated for the Node in an
+ internal map, dispatch a new BindTaskEvent, and record a `QuotaApproved` event on the Pod.
+ - In `postTaskBound()`, we update the Pod’s state to `QuotaApproved` as this will cause the default scheduler
+ to re-evaluate the pod for scheduling (more on this below).
+ - In the scheduler cache, we track pending and in-progress pod allocations, and remove them if a pod is
+ removed from the cache.
+
+## Plugin Implementation
+
+To expose the entirety of YuniKorn functionality, we implement three of the Scheduling Framework Plugins:
+
+### PreFilter
+
+PreFilter plugins are passed a reference to a Pod and return either `Success` or `Unschedulable`, depending
+on whether that pod should be considered for scheduling.
+
+For the YuniKorn implementation, we first check the Pod to see if we have an associated `applicationId`
+defined. If not, we immediately return `Success`, which allows us to delegate to the default scheduler for
+non-batch workloads.
+
+If an `applicationId` is present, then we determine if there is a pending pod allocation (meaning the
+YuniKorn core has already decided to allocate the pod). If so, we return `Success`, otherwise `Unschedulable`.
+Additionally, if an in-progress allocation is detected (indicating that we have previously attempted to
+schedule this pod), we trigger a `RejectTask` event for the YuniKorn core so that the pod will be sent back
+for scheduling later.
+
+### Filter
+
+Filter plugins are used to filter out nodes that cannot run a Pod. Only Pods which pass the PreFilter stage
+are evaluated.
+
+For the YuniKorn plugin, we follow similar logic to PreFilter, except that we also validate that the pending
+pod allocation matches the node YuniKorn chose for the pod. If the node matches, we transition the pending
+allocation to an in-progress allocation. This helps ensure that we stay in sync with the default scheduler,
+as it is possible that we allow an allocation to proceed but the bind fails for some reason.
+
+### PostBind
+
+The PostBind extension point is used informationally to notify the plugin that a pod was successfully bound.
+
+The YuniKorn implementation uses this to clean up the outstanding in-progress pod allocations.
diff --git a/versioned_docs/version-1.5.1/design/simple_preemptor.md b/versioned_docs/version-1.5.1/design/simple_preemptor.md
new file mode 100644
index 0000000000..7dbce187c7
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/simple_preemptor.md
@@ -0,0 +1,114 @@
+---
+id: simple_preemptor
+title: DaemonSet Scheduling using Simple Preemptor
+---
+
+<!--
+ * Licensed to the Apache Software Foundation (ASF) under one
+ * or more contributor license agreements. See the NOTICE file
+ * distributed with this work for additional information
+ * regarding copyright ownership. The ASF licenses this file
+ * to you under the Apache License, Version 2.0 (the
+ * "License"); you may not use this file except in compliance
+ * with the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ -->
+# Design & Implementation of Preemption for DaemonSet Pods using Simple Preemptor
+
+The simplistic approach to preempt or free up resources of running applications for DaemonSet pods. A good example of daemonset pod is fluentd logging pod which is very essential for any applicaton pod running in the node for logging.
+
+## When to start preemption?
+[YUNIKORN-1184](https://issues.apache.org/jira/browse/YUNIKORN-1184) ensures daemon set pods have been allocated properly if resources are available on the required node, otherwise, reserve the same required node so that it can be picked up to make reservation as reserved allocation (AllocatedReserved) in the next scheduling cycle. However, the whole process of modifying the reservation to reserved allocation depends on how much resources are freed up in the meantime. Duration for freein [...]
+
+By any chance, before the next run of the regular scheduling cycle (context#schedule() ), resources become available and particularly on that specific required node, then nothing needs to be done. It just moves ahead with the next steps. In case of resource constraints, unlike the regular pod reservation, other nodes cannot be tried by calling application#tryNodesNoReserve() as this demonset pod needs to run only on the specific required node. Instead, we can fork a new go routine (trigg [...]
+
+
+## How to do preemption?
+K8s does the preemption based on the pod priority. Pods with lower priority would be chosen first and so on. Proposal is not to depend on k8s for Preemption. Instead, Core should take the responsibility of finding out the list of pods that needs to be preempted, making communications to Shim and finally expecting the preempted resources to allocate to the corresponding daemonset automatically as part of the regular scheduling cycle.
+
+### Steps in trigger_preempt_workflow() go routine:
+
+##### Reservation age check (1)
+We can introduce a new Reservation age “createtime” (can be added to the reservation object) to check against the configured value of preemption_start_delay, a property to define the minimal waiting time to start the preemption process. Once reservation age exceeds this waiting time, the next step would be carried out. Otherwise, the corresponding reservation has to wait and can be processed next time.
+
+##### Get allocations from specific required Node (2)
+Get all allocations from the required node of the daemonset pod and go through the below Pre-filter pods step to filter the pods not suited for a preemption.
+
+##### Pre-filter pods to choose Victims/Candidates suited for Preemption (3)
+
+Core should filter the pods based on the following criteria:
+
+###### DaemonSet Pods
+
+All Daemonset pods should be filtered out completely irrespective of priority settings. Depending on the “requiredNode” value of pod spec, these pods can be filtered out and cannot be taken forward for the remaining process.
+
+
+
+##### Ordering Victim pods (4)
+
+###### Pods classification
+
+Once pods has been filtered out, need to classify the pods based on its types:
+
+1. Regular/Normal Pods (RP)
+2. Driver/Owner Pods (DP)
+3. Preemption Opt out Pods (OP)
+
+This classification ensures different treatment for each type of pod so that victims can be chosen among these pods in the same order. Please refer to the above diagram. It shows the 2-Dimensional array (NOTE: “Array” has been used only for documentation purposes, need to finalize the appropriate data structure) with each sub array holding pods of the same type. 1st sub array has RP’s, 2nd sub array has DP’s, 3rd sub array has OP’s and goes on.
+
+Regular/Normal Pods (RP)
+
+The regular/normal pods should be gathered and placed in the 1st sub array as these pods are given first preference for choosing the victims. In general, preempting these pods have very little impact when compared to other types/classes of pods. Hence, keeping these pods in the first subarray is the right choice
+
+Application Owner (DP)
+
+Pod acting as owner/master for other pods in the same application should be placed in the 2nd sub array because preempting those kinds of pods has a major impact when compared to Regular pods. We can select these pods by checking whether any owner reference exists between this pod and other pods. This will help prevent scenarios such as a driver pod being evicted at a very early stage when other alternatives are available for choosing the victim.
+
+Preemption Opt out (OP)
+
+Pods can be allowed to run with the Preempt opt out option. So, Pods marked with opt out should be placed in the 3rd sub array and can be used to choose victims as a last option. For now, we can use a label such as yunikorn.apache.org/allow-preemption: false for detecting those pods.
+
+
+As and when we want to introduce a new class/type of Pods, a new sub array would be created for the same and would be placed in the main array based on its significance.
+
+###### Sorting Pods
+
+Each sub array should be sorted based on the multiple criteria:
+
+1. Priority
+2. Age
+3. Resource
+
+Each sub array would be sorted priority wise, age wise and finally resource wise. The 1st sub array carrying Regular Pods has 4 pods of priority 1 and 2 pods of Priority 2. Among the 4 pods of the same priority, 3 pods are of the same age as well. Hence, again sorting resource wise really adds value and sorts them in the above shown order. Please refer to “zone”.
+
+#### Victim pods selection strategy (5)
+
+Introduce a new configuration, preemption_victim_poselection_strategy with different options (single, multiple etc) but later options act as fallback to earlier one. Defining an order for these options should be possible and upto the Administrator to place the options in an order he/she desires. Depending on the values, the whole selection strategy mechanism can decide whether a “fallback” approach among these options should be followed or not. Depending on the value, the selection strat [...]
+
+##### 1. Single Victim Pod
+
+Single Victim Pod, but resource deviation between victim pod and daemonset pod is not beyond configurable percentage. Configuring deviation with lower percentage (for example, 5% or 10%) helps prevent evicting victim pods already running with higher resource requirements. If there are many single victims found within the defined deviation %, then selection starts based on deviation % ascending order as intent is to choose the victim as close as possible to the daemonset pod resource requ [...]
+
+##### 2. Multiple Victim Pods
+
+Multiple Victim Pods, but no. of victim pods not more than configured value. This selection strategy helps to choose more than one victim, starts with the victim (resource wise descending order) and goes upto to a stage where total resource of victims meets the daemonset resource requirements but ensuring total count of victim pods not exceeding configured value.
+
+New config: preemption_victim_pods_selection_strategy
+Possible values are single,multiple (default) or multiple,single or single or multiple
+
+In case of more than one value (for ex. single,multiple), fallback would be followed as described above.
+
+#### Communicate the Pod Preemption to Shim (6)
+
+Once the list of pods has been finalized for preemption, Core can make a call to Shim for termination using notifyRMAllocationReleased (with type as TerminationType_PREEMPTED_BY_SCHEDULER). Shim can process the request as usual by making a call to K8s to delete the pod and subsequently call failTaskPodWithReasonAndMsg to notify the pod with reasons.
+
+### What happens after Preemption?
+
+Shim makes a call to K8s to delete the pod. Once k8s delete the pod, shim gets a notification from k8 and passes the information to core. This flow happens for any pod deletion and exists even today. So, even for preempted resources, we can leave it upto the regular scheduling cycle and Core-Shim communications to allocate these freed up preempted resources to the daemonset pod as node has been already reserved much earlier before the above described whole preemption workflow has begun.
\ No newline at end of file
diff --git a/versioned_docs/version-1.5.1/design/user_group_enforcement.md b/versioned_docs/version-1.5.1/design/user_group_enforcement.md
new file mode 100644
index 0000000000..dbc46f83f3
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/user_group_enforcement.md
@@ -0,0 +1,478 @@
+---
+id: user_group_resource_usage_enforcement
+title: User Based Quota Enforcement
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Introduction
+
+Enforcing resource allocation usage is currently limited to a queue.
+As part of the allocation process for an application, which owns the allocation, triggers an update of the resource usage for users and groups.
+The tracking for [user resource usage tracking in YuniKorn](user_based_resource_usage_tracking) is described and implemented independently of the enforcement.
+
+This design builds on top of the tracking design as referenced above. Like a queue a user/group could have a limit set.
+A limit can specify a resource usage quota, and an application usage limit.
+Enforcing the limit which connects the usage to the limit settings is covered in this design.
+
+## Goals
+
+* Implementation of the enforcement for user and group limits:
+ * Resource usage quota
+ * Running applications
+* Configuration processing
+ * Process limit object values from the queue configuration
+* Updates to the tracking interface to support enforcement as part of increases:
+ * Increase a user/group running applications
+ * Increase a user/group resource usage
+
+## Non Goals
+
+* User and group retrieval is part of the k8shim which is out of scope
+* Exposing usage as a metric into prometheus
+* Enforcement of queue quotas or running applications.
+* Tracking of users and or groups, see [User and Group usage tracking design doc](user_based_resource_usage_tracking)
+
+## Configuration processing
+
+The queue configuration definition has a limit object defined as [per the documentation](../user_guide/queue_config#limits).
+User and group limits are based on those objects.
+However the current configuration processing only performs a syntax check not a content check.
+
+### Generic behaviour
+
+Limits can be configured for queues at any level. Limits are applied recursively in the case of a queue limit.
+This means that a limit on the root queue is an overall limit in the cluster for the user or group.
+A root queue limit is thus also equivalent with the partition limit. While if both the root queue limit and the partition limit exist and are not equal, an error will be returned.
+
+A limit is per individual user or group. It is not a combined limit for all users or groups specified in the limit object.
+See the below limit object as an example.
+In this example both users sue and bob are allowed to run 2 applications and can each use 10 cpu and 250G of memory.
+Users and or groups not defined are not affected by this limit object.
+
+```yaml
+- limit: "example entry"
+ maxapplications: 2
+ maxresources:
+ cpu: 10
+ memory: 250G
+ users:
+ - sue
+ - bob
+```
+
+In the case that limits are configured at multiple levels in the hierarchy a fixed order of enforcement is used.
+Like with queue quotas, checks start at the leaf queue and work their way up the hierarchy finishing at the root level.
+For the limit configuration processing the same needs to be implemented and the config should be rejected if it does not comply.
+Any limit set at any level below the root must always be smaller or equal to the root limit.
+
+The checks for the limit configuration must be part of the generic configuration checks currently performed as part of the loading of the configuration.
+We cannot allow the configuration to be accepted at the queue level and later be denied at the limit level checks. A change is all or nothing, we cannot apply partial changes. Configuration checks must be made without impacting the scheduling cycles. After the change has passed the configuration check the changed configuration must then be applied in a single action.
+
+### Wildcard interpretation
+
+Wildcard can only be used as the only entry in the limit object according to the documentation.
+There can be multiple limit objects in the overall limits for a queue.
+As part of this design the interpretation of the wildcard needs to be defined.
+Wildcards interpretation is related to the ordering of the limit objects in the overall limits object.
+The usage and interpretation will be different for users and groups.
+
+In general terms: allowing a wildcard in the user or group list only as part of the last entry of the limits list.
+After the wildcard has been added for either the user or group list we do not allow a user or group list with a non wildcard entry.
+This is especially important for the group resolution. It does allow specifying a user and group wildcard with different limits set.
+In all cases, there will only be a match using the wildcard if none of the earlier limit entries match.
+
+#### User
+
+Users that are specified as part of a limit will be assigned that specific limit.
+If a limit with a wildcard is specified all users that are not already assigned a limit will be assigned the wildcard limit.
+For the user entry the wildcard sets a default quota for every user on that queue. Overridden by the named user entries.
+
+Specifying a single limit for users using the wildcard is allowed. There is no requirement to have a wildcard entry.
+
+#### Group
+
+For groups the interpretation is slightly more complex.
+As explained as part of the tracking documentation the group that the usage is tracked against must match the membership of the user.
+A wildcard group matches any group.
+
+Specifying a wildcard for the group limit sets a cumulative limit for all users in that queue.
+If there is no specific group mentioned the wildcard group limit would thus be the same as the queue limit.
+For that reason we do not allow specifying only one group limit that is using the wildcard.
+There must be at least one limit with a group name defined.
+
+The combination of one or more named group limits with a wildcard group limit is allowed.
+There is no requirement to have a wildcard entry.
+
+The hierarchy is considered bottom up, starting from the leaf up to the root.
+The first group that is specified in the queue hierarchy for limiting usage that matches a group in the _UserGroup_ object is selected for that application.
+That definition has an impact on the wildcard interpretation and the wildcard has an impact on the group selection.
+The wildcard will indirectly affect group quotas set for a specific group higher up in the hierarchy.
+This point needs clear documentation as part of the group selection algorithm and its side effects.
+
+### Example configuration
+
+An example of a set of limits. A mixture of a user and multiple groups each with a limit.
+Followed by the catch-all for both users and groups with separate resource limits.
+
+In the example below all users, except the user “sue” have the same limit set. Members of the group “development” and the group “test” both have the same limit set. All other users who are not a member of the group “development” or “test” are counted towards the cumulative group limit.
+```yaml
+limits:
+- limit: "specific user"
+ users:
+ - "sue"
+ maxresources: {memory: 25G, vcore: 5}
+- limit: "specific groups"
+ groups:
+ - "development"
+ - "test"
+ maxresources: {memory: 100G, vcore: 10}
+- limit: "user catch all"
+ users:
+ - "*"
+ maxresources: {memory: 10G, vcore: 1}
+- limit: "group catch all"
+ groups:
+ - "*"
+ maxresources: {memory: 50G, vcore: 10}
+```
+
+### Syntax checking
+
+As part of the already existing queue configuration processing the syntax for the configuration is checked.
+This check is triggered also for a reload of the file, via the config map, or the REST interface.
+Rudimentary limit object checking is implemented as part of the config check.
+
+The above described changes around the wildcards and what is allowed is not part of the configuration validation and must be added.
+
+### Quota checks
+
+No checks are performed for the limits against the queue quotas.
+This means that as part of the current checks a limit can be set that is higher than the queue would allow.
+There is also no hierarchical check in place. This means that a child could set higher limits than is possible based on the parent settings.
+
+The configuration validation processing must be extended to the limit objects.
+At each level in the configuration the resource quota assigned in a limit must be smaller than the maximum allowed at that level for the queue.
+The queue quota processing already has this builtin and the limit should be checked against the quota.
+
+The hierarchical check for limits contains a further complexity.
+A limit at one level might not apply to the same users or groups as the limit at the level below it.
+An implementation decision will need to be made on how far we want to go with the checks.
+Most, if not all, of the configurations that we do not want to allow do not break the system.
+We will not be able to check user and group interactions as we do not know the group memberships.
+The only thing we can check is the same user or group at different levels in the hierarchy.
+
+## Configuration updates and storage
+
+The configuration is read as part of the queue configuration.
+The queue configuration when processed creates, deletes or updates queues from the queue structure as appropriate.
+For limits linked to users or groups we could pre-generate the named entries but we cannot do that for wildcards.
+To support wildcards we need to at least store the configuration inside the user group manager code.
+
+The definition of the configuration store and the methods that need to be defined will be left as an implementation detail.
+It must be internal to the user group manager and will be based on the existing configuration objects.
+
+The configuration for the limits should be maintained as a separate hierarchy from the scheduling queues and the tracked users and groups.
+This is required to allow parallel processing of changes and scheduling.
+This becomes more important with an increased number of users, groups and or queues.
+
+Proposal is to pass the full configuration after it has been accepted by the system to an independent update routine for updating the affected objects in the user group manager.
+
+### Processing changes existing objects
+
+As configuration changes are based on queue changes a single queue change could trigger multiple tracker object changes.
+
+Applying the change from the configuration to an existing object in the structure of the user group manager would be initiated from the queue configuration traversal.
+All entries inside the limit are then processed.
+A comparison between the existing and new values is required to allow the correct removal of a limit via the updated configuration.
+
+If the user object does not exist the update will be a NOP, otherwise the user object will be updated by calling the setter method with the corresponding values.
+
+Wildcards will trigger an update of all objects of the specific type, user or group, for the queue they are linked to.
+These kinds of updates could be expensive which also warrants the case for comparison of the existing values.
+
+### Queue tracking
+
+The queueTracker object is the location where the limits are actually stored. The user and group objects do not contain the settings.
+
+The setter methods defined need to transparently handle the update.
+There should be no difference in behaviour to an update of existing values or setting values that do not exist.
+As an example: a leaf queue might have a value set for the maximum resources for the user.
+The new configuration does not have a value set. The setter must clear the existing value.
+Second example: a limit on a queue is not set for a group.
+The new configuration does set a limit for the group.
+This should not result in the queue(s) in the hierarchy to be created.
+If the queue exists in the hierarchy it will be updated.
+
+The following change is made to the queueTracker object.
+New fields are added to store the limit for the applications and the resources.
+New methods are added to allow updating these new values from a configuration change.
+
+```
+package user_group_manager
+
+type queueTracker struct {
+ queueName string
+ resourceUsage *Resource
+ runningApplications map[string]boolean
+ maxResourceUsage *Resource
+ maxRunningApps unit64
+
+ childQueues map[string]*queueTracker
+}
+
+(qt *queueTracker) setMaxApplications(count uint64, queuePath string)
+(qt *queueTracker) setMaxResources(resource *Resource, queuePath string)
+```
+
+An empty queue path indicates that the change needs to be applied to the current queue object. The queuePath is a fully qualified queue path that starts with “root’.
+
+### User tracking
+
+The following change is made to the userTracker object. New methods to allow updating the new limit values stored in the queue objects.
+```
+package user_group_manager
+
+(ut *userTracker) setMaxApplications(count uint64, queuePath string)
+(ut *userTracker) setMaxResources(resource *Resource, queuePath string)
+```
+
+### Group tracking
+The following change is made to the groupTracker object. New methods to allow updating the new limit values stored in the queue objects.
+
+```
+package user_group_manager
+
+(ut *groupTracker) setMaxApplications(count uint64, queuePath string)
+(ut *groupTracker) setMaxResources(resource *Resource, queuePath string)
+```
+
+## New queue to track
+
+As part of the usage tracking new objects are created for the users and or groups that are tracked.
+For users there is a change required to make sure the correct limits are set.
+This happens at two points while processing users.
+First is during the creation of a new user tracker object.
+Second is while increasing the usage of a queue which does not exist yet in the hierarchy.
+
+The logic to retrieve the configuration should be built into the creation of the queueTracker objects.
+The objects are not pre-created. They are only created as part of the _increaseResource_ call.
+The current call does not allow passing in the identity that the queue object is tracking for.
+That detail is stored in the container object: the userTracker or groupTracker object.
+
+To allow the automatic creation of the queueTracker objects the identity will need to be passed into the increaseResource call of the queueTracker object.
+The identity and type are required to find the configuration setting and apply the proper limits during the creation of the queueTracker object.
+
+```
+package user_group_manager
+
+type trackingType int
+
+const (
+ none trackingType = iota
+ user
+ group
+)
+
+(qt *queueTracker) increaseResource(queuePath, applicationID, identity string, type trackingType, usage *Resource)
+```
+
+The _trackingType_ specifies one of the two types we can track. An iota is preferred above the usage of a plain integer or boolean (user true or false) kind of setup.
+
+## Enforcement changes
+
+### Tracker interface update
+
+The current tracker does not support denying a change. For enforcement of quotas on the usage there has to be a possibility to deny a change.
+This denial can only occur on an increase of tracked resources. A decrease of the usage, i.e. a removal of an allocation because a pod has finished, can never be denied.
+
+This limits the change to support quota enforcement to a single change in the Tracker interface.
+The _IncreaseTrackedResource_ call returns the state of the increase requested.
+The return value can either be true or false. The call must only return true if the increase is allowed for both the user and the group tracked for the request.
+If either of the userTracker or groupTracker quota would be exceeded no change is made to either object and false is returned.
+
+The following change is made to the Tracker interface:
+
+```
+package user_group_manager
+
+type Tracker interface {
+ IncreaseTrackedResource(queuePath, applicationID string, usage *Resource, user *security.UserGroup) bool
+ DecreaseTrackedResource(queuePath, applicationID string, usage *Resource, removeApp bool, user *security.UserGroup)
+}
+```
+
+The change must be applied to both userTracker and groupTracker.
+Until the change is applied to both the userTracker and groupTracker, no other updates are allowed of either object.
+The increase and decrease processing must still follow the same order. Both must update the userTracker and related groupTracker objects in the same order.
+This was a requirement described in the tracking design and that has not changed.
+
+If the call to _IncreaseTrackedResource_ returns false the allocation that triggered the change needs to be abandoned.
+Any change that was made to objects inside the scheduler, like an application, queue or node, must be reversed.
+The scheduling cycle proceeds as if no allocation was made.
+
+The error return value for both the _IncreaseTrackedResource_ and the _DecreaseTrackedResource_ have been removed.
+The reason is that there is nothing that could be done by the caller to change the outcome or fail any changes if an error was returned.
+Both functions should still log the error case as it could point to a bug in the code.
+A caller should never have to pass in empty or nil parameters.
+
+### User tracking
+
+The following change is made to the userTracker method. There is no change to the structure itself for the enforcement checks:
+
+```
+package user_group_manager
+
+(ut *userTracker) increaseResource(queuePath, applicationID string, alloc *Resource) bool
+```
+### Group tracking
+
+The following change is made to the groupTracker method. There is no change to the structure itself for the enforcement checks:
+```
+package user_group_manager
+
+(gt *groupTracker) increaseResource(queuePath, applicationID string, alloc *Resource) bool
+```
+
+### Queue tracking
+
+The following change is made to the queueTracker method. There is no change to the structure itself for the enforcement checks:
+```
+package user_group_manager
+
+(qt *queueTracker) increaseResource(queuePath, applicationID, identity string, type trackingType, usage *Resource) bool
+```
+
+Note that the above change includes the change described earlier which was made to support the automatic creation of the queue objects as part of the increaseResource call.
+### Hierarchy traversal
+
+The implementation of the hierarchical check should follow the same pattern as we currently do for the queues.
+There is one major difference between the queue checks and the user and group checks.
+For the queues we do not attempt to allocate unless there is unused quota available.
+This starts at the top of the hierarchy and prevents recursing into the depth of the hierarchy towards the leaf queue.
+
+For the users and groups we have already ended up as the leaf of the structure.
+Checks will thus start at the leaf and work their way up to the root of the structure.
+A check will be performed while traversing up the hierarchy. The traversal up the tree will stop if the check fails.
+The change is applied when unwinding the traversal. This will guarantee a change is only committed when the whole hierarchy can be updated.
+
+
+
+Diagram of the quota check traversal related to the queue hierarchy.
+
+## Exposure of quota details
+
+The usage tracking information that is part of the user group manager is exposed for external consumption via REST.
+The quotas set as part of the queue objects should be exposed in the REST output.
+Exposing the quota independent of the usage makes interpretation difficult.
+
+The quota set in the queueTracker at the specific entry is the quota that is enforced on that entry.
+From a troubleshooting perspective this would be a requirement to allow an administrator to assess state.
+
+Based on the current REST api definition the data exposed in the following two endpoints will be updated to expose the limits:
+_/ws/v1/partition/\{partitionName\}/usage/users_
+_/ws/v1/partition/\{partitionName\}/usage/groups_
+
+For both endpoints we expose the full queue hierarchy. As an example below the approximate output for the users endpoint for one user:
+
+```json
+[
+ {
+ "userName": "user1",
+ "groups": {
+ "app1": "tester"
+ },
+ "queues": {
+ "queuename": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuename": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": [],
+ "maxApplications": 10,
+ "maxResources": {
+ "memory": 50000000000,
+ "vcore": 100000
+ }
+ },
+ {
+ "queuename": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app2"],
+ "children": [],
+ "maxApplications": 0,
+ "maxResources": {}
+ }
+ ],
+ "maxApplications": 10,
+ "maxResources": {}
+ }
+ }
+]
+```
+
+An example below the approximate output for the groups endpoint for one group:
+```json
+[
+ {
+ "groupName" : "tester",
+ "users": ["user1"],
+ "queues":
+ {
+ "queuename": "root",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app2"],
+ "children": [
+ {
+ "queuename": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app2"],
+ "children": [],
+ "maxApplications": 2,
+ "maxResources": {
+ "vcore": 10000
+ }
+ }
+ ],
+ "maxApplications": 0,
+ "maxResources": {}
+ }
+ }
+]
+```
+
diff --git a/versioned_docs/version-1.5.1/design/user_group_handling.md b/versioned_docs/version-1.5.1/design/user_group_handling.md
new file mode 100644
index 0000000000..57994fadfa
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/user_group_handling.md
@@ -0,0 +1,290 @@
+---
+id: user_group
+title: User/Group handling and lookup design
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Introduction
+
+Currently, user-handling in YuniKorn is handled in a relatively simple manner. If the submitted pod contains a user label the content will be set as the username. The label name is configurable and defaults to yunikorn.apache.org/username. The value is extracted in the shim code [GetUserFromPod](https://github.com/apache/yunikorn-k8shim/blob/v1.0.0/pkg/common/utils/utils.go#L220) and passed to the core. If the label is not present or no value is set the username will be set to default.
+
+However, this makes it possible for every authenticated user to freely impersonate anyone. This allows a user to circumvent quotas and other user-based restrictions inside YuniKorn. Therefore, we need a robust, secure way to make sure that users are who they say they are. This impersonation should not be confused with [Kubernetes impersonation](https://kubernetes.io/docs/reference/access-authn-authz/authentication/#user-impersonation).
+
+In the core part of YuniKorn, we also perform group resolution, similar to a unix based system or for instance Hadoop. The default implementation is a nil-lookup. It simply populates the group slice with a single entry which is the same as the username. This is not fit for purpose and was only added as a placeholder until proper user and group resolution could be added.
+
+## Goals
+
+- Define a stable way for user & group retrieval from Kubernetes
+- Assess fit of the current group resolution in the shim
+- Definition of user and group info on Kubernetes objects
+
+## Non Goals
+
+- Authorisation or Authentication as part of:
+ - YuniKorn
+ - Kubernetes
+- Changes to the way users are propagated between the shim and the core
+
+
+## Kubernetes user and group handling
+
+Dealing with how users are actually managed in Kubernetes is out of scope. What is important is the definitions that are used by Kubernetes for users. In Kubernetes, there are two kinds of accounts defined: [service and user accounts](https://kubernetes.io/docs/reference/access-authn-authz/service-accounts-admin/). User accounts are for humans. Service accounts are set up for processes which run inside a pod and want to access the API server.
+
+### User model
+
+Kubernetes, at this point in time, only manages service accounts, not users. Kubernetes does not have objects which represent user accounts. A user with valid credentials is considered authenticated.
+
+During the authentication phase the user account is mapped to a service account. For authorisation Kubernertes uses a sophisticated role-based (RBAC) or a simpler attribute-based (ABAC) model. All authorisation is linked to the service accounts, not to the user accounts. Beside the authentication there is thus no need for the user accounts. This is reflected in Kubernetes as after the authentication phase the user account is no longer tracked or available.
+
+The model used by Kubernetes thus limits what we can do to retrieve the user and or group details. All objects that are available inside the cluster to a scheduler, like YuniKorn, only have the service account information available. Getting access to the user account means that we need to intercept the information at an earlier stage.
+
+### User information as part of authentication
+
+User accounts are required during the authentication stage. Authentication in Kubernetes is configurable via plugins. For more details, see the official document on [Kubernetes authentication](https://kubernetes.io/docs/reference/access-authn-authz/authentication/). The list of supported authentication plugins, not all of these plugins will be configured and or active by default:
+- X.509 certificate
+- Static token file
+- OpenID connect tokens
+- Authentication proxy
+- Authentication webhooks ([LDAP example](https://itnext.io/implementing-ldap-authentication-for-kubernetes-732178ec2155))
+- Bootstrap token
+- Service Account Tokens
+
+
+When using test clusters created by Minikube or KIND, X509 authentication is used by default and the current context is updated with the appropriate certificates and keys, making cluster access transparent.
+
+Authentication, independent of which option is used, provides the following details:
+- Username
+- userID
+- Groups
+- Extra information
+
+Where the details are stored or how they are retrieved depends on the authentication plugin used.
+
+Kubernetes defines a special user for anonymous access, the `system:anonymous` user belonging to the `system:unauthenticated` group. Anonymous access is a security issue and should have been turned off in most clusters. This default mapping to a standardised user does provide the possibility to process authenticated and anonymous workloads in the same way.
+
+### Groups in authentication
+
+Users may belong to zero or more groups. The authentication system of Kubernetes adds one of two default groups to each user. The `system:authenticated` group is added to the list of groups for authenticated users, the `system:unauthenticated` group is added for anonymous users.
+
+For example, when generating an X509 certificate, we can define the username and organisation information like this:
+```
+openssl req -new -key alice.key \
+-out alice.csr \
+-subj "/CN=alice/O=users/O=devops"
+```
+This certificate, after being processed, provides an authenticated user with the username alice and sets the groups list to users, devops and system:authenticated. Other authentication methods allow specifying groups in their specific way. How groups are specified for each authentication option is not relevant to this design as it is encapsulated in the authentication layer and not in the exposed model.
+
+### Current findings and challenges
+
+None of the objects in Kubernetes contain the authentication information and thus the user details. After the authentication has been processed the objects inside Kubernetes either use the service account that was specified as part of the object definition or the Default service account.
+
+
+That means that the pod which is processed by YuniKorn, represented by the `k8s.io/api/core/v1/Pod` object, only has the service account as a reference.This is similar to the details available to someone using the Kubernetes tools to describe the object.
+
+The only way to retrieve the authentication information is during the admission phase of a new object into the Kubernetes system. This process is handled by the admission controllers which are part of the decision process to allow or disallow the new object to be created, modified or removed.
+
+This finding does allow YuniKorn, via its admission controller, to retrieve the authentication information of objects.
+
+
+
+
+However there are some further complications to this process. Pods are the units that the scheduler works with. Objects like deployments or jobs do not get scheduled, they in turn create a pod to be scheduled. Pods could thus be created in a number of ways.
+
+If a pod is created by a direct request, then the pod creation process contains the authentication information. This allows the YuniKorn admission controller to access and propagate the authenticated user information as part of the MutatingAdmissionWebhook.
+
+An example of a review of a direct call to create a pod with the user information shown:
+```
+INFO webhook/admission_controller.go:180 AdmissionReview {"Namespace": "default", "UID": "3f8c8df5-0272-4c85-874f-0aa1ece74fbc", "Operation": "CREATE", "UserInfo": {"username":"minikube-user","groups":["system:masters","system:authenticated"]}}
+```
+
+On the other hand, if a deployment, job or replica set is submitted, then the pod is not directly created. This also means that the user is not identifiable. The pod that YuniKorn, via the admission controller, sees is created by a controller. The user in the authentication information is not the end user that created the original object but the controller service account.
+
+An example review of a pod created as part of a Kubernetes job with the user information shown:
+
+```
+INFO webhook/admission_controller.go:177 AdmissionReview {"Namespace": "default", "UID": "be94ea46-c1d2-442a-a46a-60589d582abd", "Operation": "CREATE", "UserInfo": {"username":"system:serviceaccount:kube-system:job-controller","uid":"32abc062-eb53-4dca-9669-55251d687939","groups":["system:serviceaccounts","system:serviceaccounts:kube-system","system:authenticated"]}}
+```
+It is possible to create separate webhooks for the API objects that could result in the creation of pods. Other objects that we should watch and mutate if necessary:
+- Deployment
+- ReplicaSet
+- DaemonSet
+- StatefulSet
+- Job
+- CronJob
+- ReplicationController (deprecated)
+
+For example, adding a modification on the admission controller which instals a MutatingAdmissionWebhook for deployments, the username can be determined:
+
+```
+INFO webhook/admission_controller.go:483 Deployment validation - pass through {"User": {"username":"minikube-user","groups":["system:masters","system:authenticated"]}}
+```
+
+By intercepting the original object the submitting user information can be retrieved in the admission controller. The set of objects of interest, mentioned in the list above, all contain a pod template to specify the pod that will be created by the controller. As part of the mutating admission webhook the pod template can be updated with the user information.
+
+To confirm that this approach is viable a simple proof of concept (POC) was created. The example POC code for the current admission controller adds the currently defined yunikorn.apache.org/username label to the pod template of a deployment: [Github commit](https://github.com/pbacsko/incubator-yunikorn-k8shim/commit/478d688675a1ca98afb895672295d61b47172d19)
+
+There is an interaction with the processing of a directly created pod. Pods that are created via a controller will have the user information already set via the pod template. The admission controller processing the request should allow for the user information to be present and not overwrite or replace it with the controller service account. This case will be further defined and detailed in the chapter on [Providing user and group externally](#providing-user-and-group-externally) due to [...]
+
+## YuniKorn group retrieval
+
+As mentioned in the introduction, the YuniKorn core can retrieve a group for a given user by doing a lookup (see implementation in [usergroup.go](https://github.com/apache/yunikorn-core/blob/161c38100317ec239965e21806ebb35c6583ddcd/pkg/common/security/usergroup.go)). The lookup is pluggable and the current implementations provided are a simple OS-based and a dummy lookup.
+
+The dummy lookup mimics the simple unix OS behaviour for new users. It sets the group to the same value as the username. The dummy lookup is enabled as the default lookup.
+
+Resolving users using the OS-based resolver has limited functionality. The container that the scheduler runs in has only a base OS installed. There is no integration with an external identity provider. That means resolving groups for users running pods would normally fail as they do not exist on the OS layer that is accessible from the scheduler.
+
+The YuniKorn core only performs a group lookup if the UserGroupInformation object passed from the shim to the core does not define a list of groups. This is independent of the resolver defined in the code. By default the shim in its current implementation only passes the user name to the core. That means it always triggers the group resolution.
+
+For our purposes, both options are too simplistic. YuniKorn core should not perform any group retrieval but instead rely on the shim and the group and user information retrieved there. With the current functionality to only resolve if no group details are provided changes required should be minimal. The current default dummy resolver should be updated to not set any group and just leave the current groups. Even if that means the user is not part of any group. Based on the [Groups in auth [...]
+
+The design does not require any code changes in the core code.
+
+## Proposed user information object
+
+The admission controller has access to more detailed user information than what is available on the objects that are available to the shim. See the [Current findings and challenges](#current-findings-and-challenges) for more details.
+
+
+First change is that we need to use annotations instead of [labels](https://kubernetes.io/docs/concepts/overview/working-with-objects/labels/). The value of labels is limited to 63 characters, which will not be flexible enough to hold group information. If a user is a member of multiple groups it could easily exceed the value limit. Spitting the groups over multiple labels or encoding is not a viable option as stability is not a guarantee.
+
+Supporting a single label for the user’s name and annotations for the groups is not a good user experience. We must provide a single object that can contain the user name as well as the group details.
+
+### Annotation name
+
+The new name for the annotation should clearly cover the content and not interfere with existing annotations. Beside that we are free to use any name that complies with the naming [conventions for names](https://kubernetes.io/docs/concepts/overview/working-with-objects/annotations/#syntax-and-character-set).
+
+The proposal is to use one annotation:
+`yunikorn.apache.org/user.info`
+
+### Annotation content
+
+There is already a definition for the user information as part of the scheduler interface.The definition in the scheduler interface is:
+
+```
+message UserGroupInformation {
+// the user name
+string user = 1;
+// the list of groups of the user, can be empty
+repeated string groups = 2;
+}
+```
+
+The content of the `UserGroupInformation` message as defined in the Scheduler Interface provides a stable Go object with defined group ordering. Since the content of an annotation must be a simple string we should use a simple json representation for the user info that contains the user name and a list of groups. This also guarantees the ordering of the groups to be stable and will not change when read from the pod.
+
+
+An example of the annotation for a user called joe who is a member of the group developers. The system:authenticated group is added by the Kubernetes authentication layer:
+
+```json
+yunikorn.apache.org/user.info = "
+{
+ username: \"joe\",
+ groups: [
+ \"developers\",
+ \"system:authenticated\"
+ ]
+}"
+```
+
+### Annotation mutation
+
+In the current design a restriction on mutating the annotation is required to prevent users from updating the pod after creation. After YuniKorn is informed of the pod’s existence and the application is created the user information will not change inside YuniKorn. Kubernetes however does not guarantee that the shim will be informed of the pod creation before any modifications are made to the pod. This would leave a short window for a user to create a pod and then update the pod. Allowing [...]
+
+The admission controller must therefore validate update requests that include changes to the annotation. It is the only way to guarantee that the user information in the annotations cannot be modified.
+
+The YuniKorn admission controller is currently only registered as a mutating admission controller for pods with the operation type “create”. This can be extended to include operation type “update” for pods. A validating hook is registered for the configmap object, this one could be extended to include the pod object.
+
+This means that there are two options for the update operation:
+- mutating
+- validating
+
+In the validating case we can simply reject the change if the annotation is changed as part of the update request. In the mutating case we can undo the change by overwriting the annotation with the original value. Both solutions will work, and will take about the same amount of processing time in the admission controller.
+
+The proposal is to add the pod object to the validating webhook for update operations.
+
+### Backwards compatibility
+
+The current label will remain supported for the 1.x minor releases. Deprecation will be announced with the first release that supports the new annotation. Messages mentioning the processing of the old label will also be logged at a WARN level in the logs. The existing behaviour might require changes to the default setting see [Configuration](#configuration) for more details.
+
+Removal of processing of the currently supported label should be part of the next major release. The next major release is 2.0.0. This is based on the fact that we do not have a deprecation policy defined as yet.
+
+Preference in processing will be with the new annotations. In the case that both the user label and new annotations are present on the namespace the new annotation will be used. Using both the user label and new annotations, i.e. merging of the two, will not be supported.
+
+## Providing user and group externally
+
+As with the current label for the username we could allow for providing user and group information via annotations on the submitted objects. Although this is not a secure solution, as is pointed out in the introduction, support for this option might be a requirement to support certain use cases.
+
+We must build in an option to turn the functionality off and or mitigations to limit the possibility to bypass reading the authentication information. Based on the earlier diagram of the admission controller being used to intercept the authentication information a similar diagram for this use case. The admission controller will validate the annotation provided by the user, or system, and enforce compliance with the rules set by the administrator.
+
+
+
+
+### Pods from controllers
+
+The first use case for allowing the user information to be provided on pods when they are created is linked to controller created pods.
+
+To correctly account for these pods the admission controller should be able to not overwrite the user information if the submitting user is one of a specific group of users. Depending on the way the system is configured all controllers run as the same service account or each controller uses a specifically created service account. The [RBAC documentation](https://kubernetes.io/docs/reference/access-authn-authz/rbac/#controller-roles) describes the two options.
+
+If the controller manager runs with the option `--use-service-account-credentials` each controller uses a service account in the kube-system namespace. Running with the option set is recommended from an auditing point of view.
+
+If the option is not enabled, which seems to be the default, the controller submits the pods as the user `system:kube-controller-manager`.
+
+Both values of the option must be supported by the admission controller. That means the admission controller must have an allow list which contains at least the following users:
+- `system:serviceaccount:kube-system:*`
+- `system:kube-controller-manager`
+
+An option would be to further limit the service accounts to a subset of the service account from the kube-system namespace. The configuration should support this without changes, it would come down to a change of the default value for the allow list.
+
+There is only a limited set of controllers that can create pods. The group information does, at this point in time, not look deterministic enough to warrant a group allow list.
+
+Note that the user system:kube-controller-manager name is based on some older github issue references and needs to be confirmed in a test setup.
+
+### External applications
+
+In a similar setup applications could be used to provide a frontend for pod creation. These front-ends might or might not use individual users to submit pods. They could even leverage jobs or other objects to indirectly create pods.
+
+As an example an application might be used to provide a simple user interface to create a Spark application. Apache Airflow, Livy or similar tools come to mind.
+
+The admission controller should allow the annotation to be provided as part of the pod creation. This follows the same code and configuration path as designed for the pods submitted by the controllers.
+
+This functionality must be configurable separately from the pods created by the controllers. A similar allow list should be configurable to limit the user. For this use case we also should support an allow list for groups of users. This should allow administrators to specify one or more groups of users that are allowed to set the user information on the pods.
+
+For the first design there are no plans to add deny lists for either groups or users.
+
+## Configuration
+
+As described in the previous sections, making the behaviour configurable is a requirement. The proposal is to add the following configuration values for the behaviour. All configuration options are prefixed by “ADMISSION_CONTROLLER_”.
+
+
+The setting should be migrated to a configuration object or file when that is implemented for the admission controller. At this point the configuration uses environment settings as part of the deployment file.
+
+
+|Name|Description|Default|
+|--|--|--|
+|`BYPASS_AUTH`|Allow external users to create pods with user information set. Type: boolean |false
+|`BYPASS_CONTROLLERS`|Allow controller users to create pods with user information set. Type: boolean |true
+|`SYSTEM_USERS`|Regular expression for the allowed controller service account list. Type: string|"system:serviceaccount:kube-system:*"
+|`EXTERNAL_USERS`|Regular expression for the allowed external user list. Type: string|""
+|`EXTERNAL_GROUPS`|Regular expression for the allowed external group list. Type: string|""
+
+The configuration will be read as part of the startup of the admission controller. Automatic refresh or reload of the settings to allow changes while the admission controller is running is not part of the design as it is not supported for any other settings at this moment.
+
+If `BYPASS_AUTH` is set to true the admission controller will not add the annotation to a pod if the annotation is not present and the deprecated user labell is set. If the annotation is not set and the user label is not set the new annotation will be added. In the case that `BYPASS_AUTH` is false, the default, the admission controller will always add the new annotation, regardless of the existence of the deprecated label.
+
+The behaviour will take into account the user and group filters as defined in `EXTERNAL_USERS` and `EXTERNAL_GROUPS`. This means that retaining the existing behaviour and preferencing the existing label will require changing a default setting.
diff --git a/versioned_docs/version-1.5.1/design/user_group_manager.md b/versioned_docs/version-1.5.1/design/user_group_manager.md
new file mode 100644
index 0000000000..dab7057981
--- /dev/null
+++ b/versioned_docs/version-1.5.1/design/user_group_manager.md
@@ -0,0 +1,410 @@
+---
+id: user_based_resource_usage_tracking
+title: User Based Resource Usage Tracking
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## Introduction
+
+Tracking resource allocation usage is currently limited to a queue. An application, which owns the allocation, is the source of that resource usage. All applications that are part of the queue will count towards the resource usage of the queue. A queue could also have a resource usage quota set. The queue’s resource quota is enforced during the scheduling cycle.
+
+An application is always owned by a user/group. One user/group could own applications in one or more queues. Like a queue a user/group could have a limit set. A limit can specify a resource usage quota, and an application usage limit. Tracking these usages per user/group is currently not implemented and is covered as part of this design.
+
+## Goals
+
+- Specify tracking usage in relation to the following objects:
+ - users
+ - applications
+ - queues
+- An interface to define how to track running applications:
+ - Read a user/group running applications
+ - Increase a user/group running applications *)
+ - Decrease a user/group running applications
+
+- An interface to define how to track usage:
+ - Read a user/group resource usage
+ - Increase a user/group resource usage *)
+ - Decrease a user/group resource usage
+
+ *) This call could be extended for enforcement later.
+
+- Expose user/group tracked information through REST API
+
+## Non Goals
+- User and group retrieval is part of the k8shim which is out of scope
+- Exposing usage as a metric into prometheus
+- Specify enforcement rules
+- Implementation of enforcement
+- Interface definition to allow enforcement
+
+## Tracking definitions
+
+### User vs Group tracking
+
+Tracking is based on the _security.UserGroup_ object defined in the core scheduler. User and group information is retrieved by the shim and communicated via the _UserGroupInformation_ object as defined in the scheduler interface. A _security.UserGroup_ object contains:
+- user (exactly 1)
+- groups (0 or more)
+
+Limits can be configured for both users and groups at the same time in the same queue. So, in the case of users for which no limit has been configured explicitly but group (to which the user belongs to) limits have configured works based on the group limits configuration. In case of users for which a limit has been configured explicitly works based on its own configured limits irrespective of whether configuration has been set for the corresponding group (to which the user belongs to) o [...]
+
+"*" is also allowed for both users and groups limit configuration. In this case, any user or group needs to go through enforcement checks to ensure limit settings are being honoured properly. However, the tracker does not need to track the wildcard user as a specific user or group is already being tracked during their activities.
+
+### Group limitations
+
+Users can belong to zero, one or multiple groups. In some systems, like unix, a concept of a primary group exists. This concept is not universal and there is no consistent definition of a primary group either. If the system does not define a primary group how would we distinguish between the groups?
+
+In the case the user is a member of multiple groups the assignment of usage to a group also becomes problematic. The decision which group to assign the usage to becomes arbitrary. As an example consider a user with multiple groups assigned. Joe has groups dev and test assigned. When an application is started does that application count towards the limits for the dev group or against the limits for the test group or both?
+
+The _UserGroup_ object contains a slice, list or array, of groups which has a fixed order from the moment of creation. The group resolution configured might not guarantee an ordering which means we cannot rely on the order in the _UserGroup_ to be stable. A stable and pre-defined way of choosing a group to track usage against must be defined.
+
+The first point to define is that tracking can only occur against zero (0) or one (1) group. Usage can never be tracked against multiple groups. If a user is a member of multiple groups, a group is selected when the first usage is registered for the application. The group selected out of the list of groups in the _UserGroup_ object to track against is based on the hierarchical queue limit configuration. The hierarchy is considered bottom up, starting from the leaf up to the root. The fir [...]
+
+If no group limits are defined, in the hierarchy, usage is not tracked for a group. If no groups are defined in the _UserGroup_ object no usage will be tracked against a group.
+
+The tracker considers only the group received as part of the _UserGroup_ object at the time of first usage registration for the application in the tracker. In case of group changes while the user’s application is actively being tracked, the tracker does not consider the group change. However, If the user submits a new application, the most current group information at that point in time will be used.
+
+Until there is full support for groups in the k8shim, group tracking will be limited. No groups are passed by the k8shim to the core. This results in the _UserGroup_ object in the current implementation to always have only one group. The group name will be the same as the username. We also have excluded configuration processing for the tracking. For the current tracking implementation as described in the document these limitations will result in the user and group linked to the user to t [...]
+
+### Cluster or queue based tracking
+
+Tracking user usage will be based on the usage per queue. The full hierarchy of the queue will be tracked. We should not just track the leaf or root of the hierarchy. The overall cluster usage is the same as the usage of the root queue. Cluster level usage does not need separate tracking.
+
+The overhead of tracking per queue should be really limited. In most cases a user will run their workloads in a specific queue. Deeply nested queues do also add overhead to the tracking. During testing the impact of workload spread and or deep nesting needs to be assessed. If needed this can be called out in the documentation.
+
+We also need to take into account that multiple members of a group could be using different queues in the hierarchy. If tracking usage would be limited to the leaf level, aggregating group usage becomes more difficult and loses transparency.
+
+The documentation around limits, part of the enforcement, reference different limits at different levels for users or groups. Tracking at just the leaf or root level could break that functionality. The design should support the current documented features. If during enforcement design a different decision is made then the tracking is not the deciding factor.
+
+## Application tracking
+
+Applications go through a number of states within the core scheduler. Allocations linked to applications are the triggers for application state changes. From a tracking perspective we need to define what is considered a running application.
+
+Since placeholders and placeholder timeout can play a role in state changes the implementation of the running application tracking must take into account these cases:
+- no gang defined
+- gang defined, all or some placeholders replaced
+- gang style "soft" defined, all placeholders time out
+- gang style "hard" defined, all placeholders time out
+
+### Running state entry
+
+The application when submitted and placed into a queue is in the _New_ state. At this point there is no allocation or pending request present on the application. After one or more requests, _AllocationAsks_, are added the application moves into an _Accepted_ state. The _Accepted_ state is exited when the first _Allocation_ is added to the application. The application then transitions into the Running state.
+
+At this point a resource quota would be used by the application and the application should be considered as running from a tracking perspective. This means that the addition of the first _Allocation_ onto the application also must be the trigger point for the increase of the running applications. This trigger point for tracking is when the application is in the _Accepted_ state. This is also the point at which the group for the usage tracking needs to be set as described in the [group li [...]
+
+Note that currently, the application state transition code block in application_state.go updates the application running queue metrics when the application enters _Running_ state. The metric must be updated to be consistent with the above definition of a running application. Linking this back to a state transition the entry into the Running state should be used.
+
+### Running state exit
+
+An application should be considered running as long as at least one allocation is assigned to the application. The removal of the last allocation should decrease the number of running applications. However, we also need to consider the pending requests.
+
+As part of the application states the application enters into a _Completing_ state when the last allocation is removed and there are no pending requests. This automatically progresses into the _Completed_ state if nothing changes for the application. Applications going into the _Completing_ state all have no allocations left and pending requests are considered. The entry of the _Completing_ state thus could be used as a point for triggering the decrease of the running application count.
+
+However there are circumstances that the application goes from the _Completing_ state back into a _Running_ state. This is linked to auto restart of a pod by Kubernetes. This case must be handled.
+
+A last case that needs to be considered is an application that fails. If an application fails it moves into the Failing state. The application does not enter _Completing_ but could have incremented the running applications. A failure could be triggered by the gang scheduling style "hard" or the removal of a partition.
+
+## Resource usage tracking
+
+Resource usage is tracked simply based on the addition or removal of an allocation.
+
+Removal of an allocation is an out-of-band action. The shim notifies the core scheduler that an allocation no longer exists and the core updates the tracking based on that removal. If an allocation is preempted the core initiates the removal but still gets notified by the shim that the removal has occurred. Until the confirmation of the shim is received no tracking updates must be created.
+
+The replacement of a placeholder by the real allocation for gang scheduling follows a similar concept. However, in that case there is a possibility that there is a discrepancy between the placeholder size and the real allocation. See [YUNIKORN-982](https://issues.apache.org/jira/browse/YUNIKORN-982) for more detailed information on this issue. The changes that fixed the accounting for the queues and nodes also need to be taken into account for user based resource tracking.
+
+Adding a new allocation to an application triggers the update of the tracked resources for the user and or group as described in the [group limitations](#group-limitations). Enforcement is not a goal of the design document but the enforcement implementation needs to be taken into account. The point to add the resource usage must line up with the point that the queue structure is currently updated and the queue quotas are enforced.
+
+The order of the updates: queue structure first then user or user first then queue structure is debatable. The current proposal is to implement the following:
+- Update user and group based usage
+- Queue quota enforcement and update
+- If the queue enforcement fails, roll back the user and group based update.
+
+## Scheduler transparency
+
+Resource usage tracking is based on the allocated resource, and linked to the application’s queue path and the application’s _user_ field. Application tracking is a simple count of the applications, and is linked to the application’s queue path and the application’s user field.
+As an extension on this we could consider adding the _applicationID_ into the tracking of the running applications to help with troubleshooting. This is part of the current proposal. The proposal is to not link the running application tracking to the applicationID.
+
+The user field of an application is the earlier mentioned _security.UserGroup_ object and has all the identifying details needed to track the usage.
+
+From the schedulers perspective there should be no special setup needed. The tracking should always work without the need to pre-create a user, group or queue based entry. This means that the tracking cannot fail even if the user or group has no enforcement linked to it and does not exist in the tracker.
+
+In summary the tracker must implement:
+- User and or group creation on the fly.
+- Auto cleanup of tracking information no longer needed.
+- Configuration processing needed for enforcement.
+
+## User Group Manager (UGM)
+
+Tracking all the resources will be part of a new self-contained module in the scheduler. This means that there must be no imports from the `scheduler.objects` package. This will allow us to run the tracker separately and push updates, if needed, into a background processing queue.
+
+There will be one user group manager created per partition as part of the partition creation process. There will be no tracking of user quotas over a partition boundary.
+
+The new self-contained module, user group manager contains all specifications and code for tracking and can be extended in the future to include enforcement code.
+
+Tracking the resources and application count for all users occurs from startup. This happens irrespective of whether enforcement is configured for the user or group. By starting tracking independently of the enforcement changes in enforcement will always have a consistent view of the resources tracked at least at a user level. Recovery must be tested in combination with the tracking code.
+
+Storing data in the user group manager is based on the _security.UserGroup_ structure content. The built-in _UserGroupCache_ resolution cannot guarantee that the same object is returned for the same user each time.
+
+### UserGroup cache impact
+
+The concept of a stable user entry in the cache is not currently present in the _UserGroupCache_. This concept will most likely also not be possible based on the fact that group resolution does not guarantee ordering. Based on the assumption that we cannot guarantee group ordering for all group resolutions that are possible the tracking will not rely on it.
+
+Currently, _UserGroupCache_ holds the _UserGroup_ object entry only for the cache period. This is either 300 seconds for a positive resolution or 30 seconds for a negative resolution. After this cache time the entry will be deleted as part of the cache cleanup/expiration process. If the same user submits a second application after the expiry, a new _UserGroup_ object will be created and stored in the cache. The _UserGroup_ object has a property "resolved", which holds a timestamp at whic [...]
+
+Other internal changes might be needed to make sure all _UserGroup_ objects are cached correctly and returned as expected internally in the cache. These are left as implementation details for the cache changes.
+
+### Tracking structure
+
+The user group Manager must be able to track usage for users and groups. Users are the primary access point for all tracking. The user group Manager structure will implement the [tracker interface](#tracker-interface).
+
+````
+package user_group_manager
+
+type Manager struct {
+ users map[string]*userTracker
+ groups map[string]*groupTracker
+
+ sync.RWMutex
+}
+(m *Manager) checkAppGroup(queuePath, applicationID string, user *security.UserGroup)
+````
+
+All objects in the user group manager will be managed internally. Creation of user and or group tracker objects is based on the request to track usage. The objects are added to one of the two structures in the user group manager. Adding user or group objects can only be triggered by an increase of tracked resources. The creation of the user and or group tracker objects is handled by the user group manager. The user group manager is also responsible for setting the usage information for t [...]
+
+The resolution of the group that the usage is tracked against for an application is handled by the user group Manager via the checkAppGroup. As part of an increase of resource usage the Manager will check if the application is already tracked for the user. If the application is already tracked nothing changes. If the application is not tracked the manager will resolve the group to track against and update the userTracker object. If there is no active tracker registered for the group a ne [...]
+
+Removal of user or group objects can only be triggered after a decrease of tracked resources has removed the last usage. Usage tracked for a user must be zero. For group tracker objects the application reference count must be zero.
+Removal of the user and or group tracker objects is done periodically. This cleanup must be `go routine` safe. Updates to either map could be triggered from different routines, scheduler allocation processing or a release of an allocation by a shim.
+
+### Tracker interface
+
+The interface describes the functions that will be exposed by the user group manager. The other scheduler components can rely on the functions defined in the interface to be available to increase, decrease applications and or resource usage and get the tracked resources.
+
+```
+package user_group_manager
+
+type Tracker interface {
+ GetUserResources(user *security.UserGroup) *Resource
+ GetGroupResources(group string) *Resource
+
+ GetUsersResources() []*userTracker
+ GetGroupsResources() []*groupTracker
+
+ IncreaseTrackedResource(queuePath, applicationID string, usage *Resource, user *security.UserGroup) error
+ DecreaseTrackedResource(queuePath, applicationID string, usage *Resource, removeApp bool, user *security.UserGroup) error
+}
+```
+
+The _IncreaseTrackedResource()_ and _DecreaseTrackedResource()_ calls will implement the tracking data updates. The Get* calls are providing safe access to the tracked data to be used as part of exposing the data in the REST interface. See [exposure of usage tracking](#exposure-of-usage-tracking) for more detail.
+
+Applications are added automatically during the call to increase the tracked resources. The first time the applicationID is encountered the structures are updated accordingly and tracking starts. The removal of the application is triggered by leaving the [running state](#running-state-exit), this is equivalent to the removal of the last usage and must also set the removeApp flag to true.
+
+Calling _IncreaseTrackedResource()_ and _DecreaseTrackedResource()_ with empty or nil parameters is not allowed and will return an error to be handled by the caller. The _GetUserResource_ and _GetGroupResource_ calls can return a _nil_ Resource value. The _GetUsersResource_ and _GetGroupsResource_ calls can return an empty slice.
+
+### User tracking
+
+The userTracker contains all tracked usage information for a user. Changes for all usage, even the group usage, are part of the change that gets triggered on the user object.
+
+The userTracker object contains a reference map to the usage for each application that is tracked linking it to the groupTracker object for the specific application. Multiple applications can be tracked against the same groupTracker object.
+
+The group references will be set for each application when usage is added for an application and is not mutable. The existence of a groupTracker for the application can be checked by calling _hasGroupForApp_ on the userTracker object. It returns true in case the application ID exists in the group mapping, false if it does not. The return value is only based on the existence of the applicationID as the key, not on the value it references.
+
+The groupTracker can be set in the userTracker object by calling _setGroupForApp_. If the applicationID already exists, the call to set the groupTracker reference is a noop, nothing will change. The reference cannot be changed after it is set. The groupTracker reference can be nil. In the case of a nil reference, no tracking for that application against a group is performed. Even a nil reference cannot be replaced.
+
+An increase or decrease of the resources tracked for the user requires the applicationID to be set to a non-empty value. On removal of the last allocation for the application the group reference will be removed from the userTracker object. To remove a running application from the userTracker the applicationID must be set in the _decreaseResource_ call and the flag removeApp must be set to true. Removal of the groupTracker reference can only occur as part of a _decreaseResource_ call.
+
+If there are no applications left in the group references the object will be removed by the cleanup run from the user group manager. This will also mean that there is no usage tracked anymore by the referenced queueTracker object.
+
+The usage per queue is stored in a queue structure specific for this user. The queueTracker object provides a hierarchical structure. This structure starts at the root of the queue hierarchy and only contains child queues for which a usage is being tracked. The hierarchy must be updated, add or delete of child entries, as part of the updates that are processed.
+
+The creation of the root queueTracker is part of the creating a new userTracker object.
+
+```
+package user_group_manager
+
+type userTracker struct {
+ userName string
+ group map[string]*groupTracker
+ queue *queueTracker
+
+ sync.RWMutex
+}
+
+newUserTracker(queuePath string, user *security.UserGroup) *userTracker
+
+(ut *userTracker) increaseResource(queuePath, applicationID string, usage *Resource)
+(ut *userTracker) decreaseResource(queuePath, applicationID string, usage *Resource, removeApp bool)
+(ut *userTracker) hasGroupForApp(applicationID string) bool
+(ut *userTracker) setGroupForApp(applicationID string, groupTrack *groupTracker)
+```
+
+### Group tracking
+
+The groupTracker contains all tracked usage information for a group. This object can be shared by multiple users for one or more applications. It adds the applications that are tracked for this specific group automatically to the list. For this it expects an applicationID to be set to a non-empty value for each _increaseResource_ call.
+
+To simply decrease the resources tracked for the group the applicationID could be empty. On removal of the last allocation for the application the application will be removed from the groupTracker object. To remove a running application from the groupTracker the applicationID must be set in the _decreaseResource_ call and the flag removeApp must be set to true.
+
+If there are no entries left in the applications tracked in the groupTracker the object will be removed by the cleanup run from the user group manager. This will also mean that there is no usage tracked any more by the referenced queueTracker object.
+
+The usage per queue is stored in a queue structure specific for this group. The queueTracker object provides a hierarchical structure. This structure starts at the root of the queue hierarchy and only contains child queues for which a usage is being tracked. The hierarchy must be updated, add or delete of child entries, as part of the updates that are processed.
+
+The creation of the root queueTracker is part of the creating a new groupTracker object.
+
+```
+package user_group_manager
+
+type groupTracker struct {
+ groupName string
+ applications map[string]bool
+ queue *queueTracker
+
+ sync.RWMutex
+}
+
+newGroupTracker(queuePath string, group string) *groupTracker
+
+(gt *groupTracker) increaseResource(queuePath, applicationID string, usage *Resource)
+(gt *groupTracker) decreaseResource(queuePath, applicationID string, usage *Resource, removeApp bool)
+```
+
+### Queue tracking
+
+The queueTracker acts as a base for both user and group tracking. It provides the actual location for the tracked usage.The top of the structure, which is referenced from the userTracker or groupTracker objects, contains the root queue usage. The queueTracker is only referenced by one userTracker or one groupTracker object and must not be shared. This means that we should not require any locking within the queueTracker object.
+
+The queue hierarchy below the root, for which a usage is registered, is part of the childQueues. Each part of the full queue path creates a new level of objects. This structure shows a sparse hierarchical queue structure. It may not represent the full queue structure as known in the scheduler.
+
+The creation of the root queueTracker returns a plain object without usage set and a fixed queueName of "root". The usage should then be updated by a call to _increaseResource_. The call must handle the recursive creation of the queue path as given. It expects an applicationID to be set to a non-empty value for each call.
+
+To simply decrease the resources tracked for the queue the applicationID can be empty. On removal of the last allocation for the application the application will be removed from the queueTracker. To remove a running application from the queueTracker the applicationID must be set in the _decreaseResource_ call and the flag removeApp must be set to true.
+
+```
+package user_group_manager
+
+type queueTracker struct {
+ queueName string
+ resourceUsage *Resource
+ runningApplications map[string]boolean
+
+ childQueues map[string]*queueTracker
+}
+
+newQueueTracker() *queueTracker
+
+(qt *queueTracker) increaseResource(queuePath, applicationID string, usage *Resource)
+(qt *queueTracker) decreaseResource(queuePath, applicationID string, usage *Resource, removeApp bool)
+```
+
+### Update consistency
+
+All changes are triggered against a userTracker object. Since enforcement of a quota is out of scope of this design we can ignore update failures due to quota exhaustion. There is no requirement or possibility that updates need to be rolled back.
+
+Usage updates can be triggered from two points: allocation by the scheduler and releases by a shim. Allocations as well as releases need to update both user and group usage information. We must guarantee that for both the full hierarchy is changed or neither are changed.
+
+The releases are also processed asynchronously from the scheduling cycle. This means that we have to be able to guarantee that the increases and decreases are applied consistently to both objects also. Locking of the objects to serialise changes must be in place to support this.
+
+The other point is the update ordering and locking. The increase and decrease processing must follow the same order. Both must update the userTracker and related groupTracker objects in the same order. This is a requirement to prevent the processing of being able to deadlock while updating the same user for a simultaneous increase and decrease.
+
+For groupTracker updates that originate from two different users the locking of the groupTracker object is required. One of the two updates will be able to proceed while the other waits for the update to finalise.
+
+A final consideration is the consistency of two updates on the same object that follow each other closely. If the first update removes all existing usage the userTracker or groupTracker object is cleared for removal. While the update is in progress a second update comes in to add a new usage to the same object. The resolution of the userTracker in the user group manager has finished. The code has a reference to the object. Processing is waiting to lock the userTracker object. This point [...]
+
+### Exposure of usage tracking
+
+The usage tracking information that is part of the user group manager must be exposed for external consumption. All our current information is exposed via a REST call.
+
+Based on the current REST api definition for other object the proposal is to expose the following new endpoints:
+
+- /ws/v1/partition/\{partitionName\}/usage/users
+- /ws/v1/partition/\{partitionName\}/usage/groups
+
+For both endpoints we expose the full queue hierarchy. As an example below the approximate output for the user's endpoint for one user:
+
+```json
+[
+ {
+ "userName": "user1",
+ "groups": {
+ "app2": "tester"
+ },
+ "queues":
+ {
+ "queuename": "root",
+ "resourceUsage": {
+ "memory": 12000000000,
+ "vcore": 12000
+ },
+ "runningApplications": ["app1", "app2"],
+ "children": [
+ {
+ "queuename": "root.default",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app1"],
+ "children": []
+ },
+ {
+ "queuename": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": [
+ "app2"
+ ],
+ "children": []
+ }]
+ }
+ }
+]
+```
+
+
+An example below the approximate output for the groups endpoint for one group:
+
+```json
+[
+ {
+ "groupName" : "tester",
+ "applications": ["app2"],
+ "queues":
+ {
+ "queuename": "root",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app2"],
+ "children": [
+ {
+ "queuename": "root.test",
+ "resourceUsage": {
+ "memory": 6000000000,
+ "vcore": 6000
+ },
+ "runningApplications": ["app2"],
+ "children": []
+ }
+ ]
+ }
+ }
+]
+```
diff --git a/versioned_docs/version-1.5.1/developer_guide/build.md b/versioned_docs/version-1.5.1/developer_guide/build.md
new file mode 100644
index 0000000000..55f71e895a
--- /dev/null
+++ b/versioned_docs/version-1.5.1/developer_guide/build.md
@@ -0,0 +1,244 @@
+---
+id: build
+title: Build and Run
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+YuniKorn always works with a container orchestrator system. Currently, a
+Kubernetes shim ([yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim))
+is provided which provides a drop-in scheduler for the Kubernetes platform.
+This document describes how to setup and use a local development environment.
+
+## Dev Environment setup
+
+Read the [Dev Environment Setup](developer_guide/env_setup.md) guide first to
+setup Docker and Kubernetes development environment.
+
+## Build YuniKorn
+
+Prerequisites:
+- Golang: check the `.go_version` file in the root of the repositories for the
+version Yunikorn requires. The minimum version can change per release branch.
+Using earlier Go versions will cause compilation issues.
+
+You can build the scheduler for Kubernetes from the [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim)
+project. The build procedure will build all components into a single executable
+that can be deployed and running on Kubernetes.
+
+Start the integrated build process by pulling the `yunikorn-k8shim` repository:
+```bash
+mkdir $HOME/yunikorn/
+cd $HOME/yunikorn/
+git clone https://github.com/apache/yunikorn-k8shim.git
+```
+
+At this point you have an environment that will allow you to build an
+integrated image for the YuniKorn scheduler.
+
+### Build Docker images
+
+Building the Docker images can be triggered by following command:
+```shell script
+make image
+```
+
+This will generate images for the scheduler, scheduler plugin, and admission
+controller.
+
+The images created can be deployed directly on Kubernetes.
+Some sample deployments that can be used are found under the
+[deployments/scheduler](https://github.com/apache/yunikorn-k8shim/tree/master/deployments/scheduler)
+directory of the `yunikorn-k8shim` repository. Alternatively, the Helm charts
+located within the [helm-charts](https://github.com/apache/yunikorn-release/tree/master/helm-charts)
+directory of the `yunikorn-release` repository may be used. These match what is used
+for release builds.
+
+The configuration of YuniKorn can be customized via a ConfigMap as explained in the
+[scheduler configuration deployment](developer_guide/deployment.md) document.
+
+The `make image` build command will first build the integrated executables and
+then create the docker images. If you want to use pre-built images based on an
+offical release, please check the [Docker Hub repo](https://hub.docker.com/r/apache/yunikorn).
+
+The default image tags are not suitable for deployments to a private
+repository as these would attempt to push to Docker Hub without proper
+credentials. You *must* update the `REGISTRY` variable in the `Makefile` to
+push to an accessible repository. When you update the image tag be aware that
+the deployment examples given will also need to be updated to reflect the same
+change.
+
+### Inspect Docker images
+
+The Docker images built from previous step have embedded some important build
+info in the image metadata. You can retrieve this information with docker
+`inspect` command:
+
+```shell script
+docker inspect apache/yunikorn:scheduler-amd64-latest
+docker inspect apache/yunikorn:scheduler-plugin-amd64-latest
+docker inspect apache/yunikorn:admission-controller-amd64-latest
+```
+
+The `amd64` tag is dependent on your host architecture (i.e. for Intel it would
+be `amd64` and for Mac M1, it would be `arm64`).
+
+This info includes git revisions (last commit SHA) for each component, to help
+you understand which version of the source code was shipped by this image. They
+are listed as docker image `labels`, such as
+
+```json
+"Labels": {
+ "BuildTimeStamp": "2019-07-16T23:08:06+0800",
+ "Version": "0.1",
+ "yunikorn-core-revision": "dca66c7e5a9e",
+ "yunikorn-k8shim-revision": "bed60f720b28",
+ "yunikorn-scheduler-interface-revision": "3df392eded1f"
+}
+```
+
+### Dependencies
+
+The dependencies in the projects are managed using
+[go modules](https://blog.golang.org/using-go-modules).
+
+If you want to modify one of the projects locally and build with your local
+dependencies you will need to change the module file. Changing dependencies
+requires using `go.mod` `replace` directives as explained in the
+[Update dependencies](#updating-dependencies) section.
+
+The YuniKorn project has four code repositories:
+ - [yunikorn-scheduler-interface](https://github.com/apache/yunikorn-scheduler-interface)
+ (protobuf interface between core and shim)
+ - [yunikorn-core](https://github.com/apache/yunikorn-core)
+ (core scheduler logic)
+ - [yunikorn-k8shim](https://github.com/apache/yunikorn-k8shim)
+ (Kubernetes-specific shim)
+ - [yunikorn-web](https://github.com/apache/yunikorn-web)
+ (YuniKorn Web UI)
+
+Each of these dependencies is a Go module and there are dependencies between
+them. During the development cycle it can be required to break the dependency
+on the committed version from github. This requires making changes in the module
+file to allow loading a local copy or a forked copy from a different repository.
+
+Additionally, there are two additional auxiliary repositories:
+ - [yunikorn-release](https://github.com/apache/yunikorn-release)
+ (release management scripts and official Helm charts)
+ - [yunikorn-site](https://github.com/apache/yunikorn-site)
+ (source of the yunikorn.apache.org web site)
+
+#### Affected repositories
+The following dependencies exist between the repositories:
+
+| Repository | Depends on |
+|------------------------------|---------------------------------------------|
+| yunikorn-core | yunikorn-scheduler-interface |
+| yunikorn-k8shim | yunikorn-scheduler-interface, yunikorn-core |
+| yunikorn-scheduler-interface | none |
+| yunikorn-web | none |
+
+The `yunikorn-web` repository has no direct go dependency on the other
+repositories. However any change to the `yunikorn-core` web services can affect
+the web interface.
+
+#### Making local changes
+
+To make sure that the local changes will not break other parts of the
+build you should run:
+- A full build `make` (build target depends on the repository)
+- A full unit test run `make test`
+
+Any test failures should be fixed before proceeding.
+
+#### Updating dependencies
+
+The simplest way is to use the `replace` directive in the module file.
+The `replace` directive allows you to override the import path with a new
+(local) path. There is no need to change any of the imports in the source code.
+The change must be made in the `go.mod` file of the repository that has the
+dependency.
+
+Using `replace` to use of a forked dependency, such as:
+```
+replace github.com/apache/yunikorn-core => example.com/some/forked-yunikorn
+```
+
+There is no requirement to fork and create a new repository. If you do not have
+a repository you can use a local checked out copy too.
+
+Using `replace` to use of a local directory as a dependency:
+```
+replace github.com/apache/yunikorn-core => /User/example/local/checked-out-yunikorn
+```
+
+For the same dependency using a relative path:
+```
+replace github.com/apache/yunikorn-core => ../checked-out-yunikorn
+```
+Note: if the `replace` directive is using a local filesystem path, then the target
+must have a `go.mod` file at that location.
+
+Further details can be found on the Go Wiki:
+[When should I use the 'replace' directive?](https://github.com/golang/go/wiki/Modules#when-should-i-use-the-replace-directive)
+
+## Build the Web UI
+
+Example deployments reference the
+[YuniKorn Web UI](https://github.com/apache/yunikorn-web). The `yunikorn-web`
+project has specific requirements for the build. Follow the steps in the
+[README](https://github.com/apache/yunikorn-web/blob/master/README.md) to prepare
+a development environment and build the Web UI. However, the scheduler is fully
+functional without the Web UI.
+
+## Run YuniKorn locally
+
+When you have a local development environment setup you can run the scheduler
+in your local Kubernetes environment. This has been tested in a desktop
+enviornment with Docker Desktop, Minikube, and Kind. See the
+[Dev Environment Setup](developer_guide/env_setup.md) guide for further details.
+
+To run a local instance of the scheduler:
+
+```shell script
+make run
+```
+
+This will launch a local scheduler and connect to the Kubernetes cluster
+referenced in your `KUBECONFIG` or `$HOME/.kube/config`.
+
+To run YuniKorn in Kubernetes scheduler plugin mode instead, execute:
+
+```
+make run_plugin
+```
+
+You can also use the same approach to run the scheduler locally but connecting
+to a remote kubernetes cluster, as long as the `$HOME/.kube/config` file
+is pointing to that remote cluster.
+
+## Run end-to-end tests
+
+In addition to the unit tests for each project, YuniKorn contains many e2e
+(end-to-end) tests in the `yunikorn-k8shim` repository which validate
+functionaliy of the scheduler on a functioning Kubernetes cluster.
+
+How to run the tests locally is described
+[here](https://github.com/apache/yunikorn-k8shim/blob/master/test/e2e/README.md).
diff --git a/versioned_docs/version-1.5.1/developer_guide/dependencies.md b/versioned_docs/version-1.5.1/developer_guide/dependencies.md
new file mode 100644
index 0000000000..d458798df2
--- /dev/null
+++ b/versioned_docs/version-1.5.1/developer_guide/dependencies.md
@@ -0,0 +1,124 @@
+---
+id: dependencies
+title: Go module updates
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+## When to update
+The references in the `master` branches must be updated if a change is made in the scheduler interface.
+Updating the dependency of a shim in reference to the core might be needed even if the scheduler interface does not change.
+New functionality could be added that rely on changed field content of the messages, not the field layout of the message.
+In that case just the shim dependency needs to be updated.
+
+## Why a pseudo version
+In the `master` branch we **must** use a pseudo version for all the YuniKorn repository references we use.
+As the branch is in active development and not released we do not have a real version tag to reference.
+However, we still need to be able to point to the right commit for the dependencies.
+
+Go allows using [pseudo versions](https://go.dev/ref/mod#pseudo-versions) for these specific cases.
+An example of the pseudo versions we use in the Kubernetes shim:
+```
+module github.com/apache/yunikorn-k8shim
+
+go 1.16
+
+require (
+ github.com/apache/yunikorn-core v0.0.0-20220325135453-73d55282f052
+ github.com/apache/yunikorn-scheduler-interface v0.0.0-20220325134135-4a644b388bc4
+ ...
+)
+```
+Release branches **must** not use pseudo versions.
+During the creation of a release, [tags](/community/release_procedure#tag-and-update-release-for-version) will be created.
+These tags will be used as the reference in the go.mod files for the release.
+
+## Enforcement of pseudo version
+In the pull request checks for the `yunikorn-core` and `yunikorn-k8shim` repositories enforce the format of the versions.
+A build failure will be triggered if the version reference for the `yunikorn-core` or `yunikorn-scheduler-interface`
+repositories in the `master` branch is not a pseudo version.
+
+The check enforces that the start of the version reference is `v.0.0.0-`
+
+Pseudo versions are not enforced in the release branches as per [why a pseudo version](#why-a-pseudo-version) explanation above.
+
+## Updating the core dependency
+Before updating the core dependency must make sure that the scheduler interface changes are finalised.
+
+1. Make the changes in the scheduler interface.
+2. Commit the changes into the master branch on GitHub and pull the latest master branch commit.
+3. [Generate a new pseudo version](#generating-a-pseudo-version) for the scheduler-interface.
+
+Updating the core dependency
+
+4. Update the go.mod file for the dependent repository: core repository
+ * Open the go.mod file
+ * Copy the generated pseudo version reference
+ * Replace the scheduler-interface version reference with the one generated in step 3.
+ * Save the go.mod file
+5. Run a `make test` to be sure that the change works. The build will pull down the new dependency and the change in the scheduler interface will be used.
+6. Commit the changes into the master branch on GitHub and pull the latest master branch commit
+
+## Updating a shim dependency
+Before updating a shim dependency you must make sure that the core dependency has been updated and committed.
+There are cases that the reference for the scheduler-interface has not changed.
+This is not an issue, either skip the update steps or execute them as per normal resulting in no changes as part of the commit.
+
+7. [Generate a new pseudo version](#generating-a-pseudo-version) for the core
+8. Update the go.mod file for the dependent repository: k8shim repository
+ * Open the go.mod file
+ * Copy the generated pseudo version reference of the scheduler interface
+ * Replace the scheduler-interface version reference with the one generated in step 3.
+ * Copy the generated pseudo version reference of the core
+ * Replace the core version reference with the one generated in step 7.
+ * Save the go.mod file
+9. Run a `make test` to be sure that the change works. The build will pull down the new dependency and the changes in the core and scheduler interface will be used.
+10. Commit the changes into the master branch on GitHub
+
+:::note
+If multiple PRs are being worked on in the scheduler interface and or core at the same time a different PR might have already applied the update.
+This will all depend on the commit order.
+It is therefor that steps 5 and 8 are performed to make sure there is no regression.
+:::
+## Generating a pseudo version
+
+A pseudo references for use in a go.mod file is based on the commit hash and timestamp.
+It is simple to generate one using the following steps:
+
+1. Change to the repository for which the new pseudo version needs to be generated.
+2. Update the local checked out code for the master branch to get the latest commits
+```
+git pull; git status
+```
+The status should show up to date with the `origin` from where it was cloned.
+3. Run the following command to get the pseudo version:
+```
+TZ=UTC git --no-pager show --quiet --abbrev=12 --date='format-local:%Y%m%d%H%M%S' --format='v0.0.0-%cd-%h'
+```
+4. This command will print a line like this:
+```
+v0.0.0-20220318052402-b3dfd0d2adaa
+```
+That is the pseudo version that can be used in the go.mod files.
+
+:::note
+The pseudo version must be based on a commit that is in the vcs system, i.e. from Github.
+Local commits or commits that are not yet merged in a PR cannot be used.
+:::
diff --git a/versioned_docs/version-1.5.1/developer_guide/deployment.md b/versioned_docs/version-1.5.1/developer_guide/deployment.md
new file mode 100644
index 0000000000..50ad30b3aa
--- /dev/null
+++ b/versioned_docs/version-1.5.1/developer_guide/deployment.md
@@ -0,0 +1,145 @@
+---
+id: deployment
+title: Deploy to Kubernetes
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+The easiest way to deploy YuniKorn is to leverage our [helm charts](https://hub.helm.sh/charts/yunikorn/yunikorn),
+you can find the guide [here](get_started/get_started.md). This document describes the manual process to deploy YuniKorn
+scheduler and admission controller. It is primarily intended for developers.
+
+**Note** The primary source of deployment information is the Helm chart, which can be found at [yunikorn-release](https://github.com/apache/yunikorn-release/). Manual deployment may lead to out-of-sync configurations, see [deployments/scheduler](https://github.com/apache/yunikorn-k8shim/tree/master/deployments/scheduler)
+
+## Build docker image
+
+Under project root of the `yunikorn-k8shim`, run the command to build an image using the map for the configuration:
+```
+make image
+```
+
+This command will build an image. The image will be tagged with a default version, image tag and your build architecture.
+
+**Note** the default build uses a hardcoded user and tag. You *must* update the `IMAGE_TAG` variable in the `Makefile` to push to an appropriate repository.
+
+**Note** the latest yunikorn images in docker hub are not updated anymore due to ASF policy. Hence, you should build both scheduler image and web image locally before deploying them.
+
+**Note** the imaging tagging includes your build architecture. For Intel, it would be `amd64` and for Mac M1, it would be `arm64`.
+
+## Setup RBAC for Scheduler
+In the example, RBAC are configured for the yuniKorn namespace.
+The first step is to create the RBAC role for the scheduler, see [yunikorn-rbac.yaml](https://github.com/apache/yunikorn-k8shim/blob/master/deployments/scheduler/yunikorn-rbac.yaml)
+```
+kubectl create -f deployments/scheduler/yunikorn-rbac.yaml
... 8469 lines suppressed ...
---------------------------------------------------------------------
To unsubscribe, e-mail: issues-unsubscribe@yunikorn.apache.org
For additional commands, e-mail: issues-help@yunikorn.apache.org