[jira] [Commented] (YARN-10496) [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler

["Wangda Tan (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17235683#comment-17235683 ] 

Wangda Tan commented on YARN-10496:
-----------------------------------

Worked with [~bteke] for a design doc, see the linked doc. Would like to see more comments from the community:

cc: [~epayne], [~jhung], [~tangzhankun], [~bilwa_st]

> [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
> ---------------------------------------------------------------------
>
>                 Key: YARN-10496
>                 URL: https://issues.apache.org/jira/browse/YARN-10496
>             Project: Hadoop YARN
>          Issue Type: New Feature
>          Components: capacity scheduler
>            Reporter: Wangda Tan
>            Priority: Major
>
> CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints: 
>  * Only leaf queues can be auto-created
>  * A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
>  * It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.
>  
>  * In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
>  ** This implies that there is no possibility to have both dynamically created and static queues at the same time under root
>  * A new queue needs to be created under an existing parent, while the parent already has static queues
>  * Nested queue mapping policy, like in the following example: 
> |<rule name="nestedUserQueue" create=”true”>
>         <rule name="primaryGroup" create="true" />
> </rule>|
>  * Here two levels of queues may need to be created 
> If an application belongs to user _alice_ (who has the primary_group of _engineering_), the scheduler checks whether _root.engineering_ exists, if it doesn’t,  it’ll be created. Then scheduler checks whether _root.engineering.alice_ exists, and creates it if it doesn't.
>  
> When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org