[GitHub] [hadoop] brumi1024 commented on a change in pull request #3330: YARN-10522. Document for Flexible Auto Queue Creation in Capacity Scheduler.

[GitBox <gi...@apache.org>]

brumi1024 commented on a change in pull request #3330:
URL: https://github.com/apache/hadoop/pull/3330#discussion_r698356952



##########
File path: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-site/src/site/markdown/CapacityScheduler.md
##########
@@ -681,6 +689,55 @@ list of current scheduling edit policies as a comma separated string in `yarn.re
 |:---- |:---- |
 | `yarn.resourcemanager.monitor.capacity.queue-management.monitoring-interval` | Time in milliseconds between invocations of this QueueManagementDynamicEditPolicy policy. Default value is 1500 |
 
+* Parent queue configuration for **flexible** dynamic leaf queue auto-creation and management
+
+The `Flexible Dynamic Queue Auto-Creation and Management` feature allows a **ParentQueue** to auto-create both parent and leaf queues. Such parent queues can have other pre-configured queues co-existing with the auto-created queues. The auto-created queues will have weights as *capacity* so the pre-configured queues under the parent must be configured the same way. The `CapacityScheduler` supports the following parameters to enable auto-creation of queues
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.enabled` | *Mandatory* parameter: Indicates to the `CapacityScheduler` that flexible auto queue creation needs to be enabled for the specified parent queue.  Boolean value expected. The default value is *false*, i.e. auto leaf queue creation is not enabled in *ParentQueue* by default. |
+| `yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.max-queues` | *Optional* parameter: Limits the number of dynamic queues created under a parent queue.  Integer value expected. The default value is *1000*. |
+
+
+* Configuring **flexible** `Auto-Created Leaf Queues` with `CapacityScheduler`
+
+The parent queue which has the flexible auto queue creation enabled supports the configuration of dynamically created leaf and parent queues through template parameters. The auto-created queues support all of the leaf queue configuration parameters except for **Queue ACL**, **Absolute Resource** configurations. Queue ACLs are currently inherited from the parent queue i.e they are not configurable on the leaf queue template
+
+| Property | Description |
+|:---- |:---- |
+| `yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.template.<queue-property>` | *Optional* parameter: Specifies a queue property (like capacity, maximum-capacity, user-limit-factor, maximum-am-resource-percent ...  - Refer **Queue Properties** section) inherited by the auto-created **parent** and **leaf** queues. |
+| `yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.leaf-template.<queue-property>` | *Optional* parameter: Specifies a queue property inherited by auto-created **leaf** queues. |
+| `yarn.scheduler.capacity.<queue-path>.auto-queue-creation-v2.parent-template.<queue-property>` |  *Optional* parameter: Specifies a queue property inherited by auto-created **parent** queues. |
+
+Using the following example configuration snippet will instruct the `CapacityScheduler` to:
+* enable the flexible auto queue creation for root.parent
+* create **all** of the dynamic queues below root.parent with 80% as the maximum capacity, because of the wildcard queue path (root.parent.*)

Review comment:
       This line is for the property yarn.scheduler.capacity.root.parent.*.auto-queue-creation-v2.template.maximum-capacity, which has a simple template tag, not the parent-template. Doesn't this apply to all of the queues, not just the parents?
   
   Or do you mean that this overrides the static queues as well?




-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org



---------------------------------------------------------------------
To unsubscribe, e-mail: common-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: common-issues-help@hadoop.apache.org