You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Peter Bacsko (Jira)" <ji...@apache.org> on 2020/12/02 11:43:00 UTC
[jira] [Comment Edited] (YARN-10496) [Umbrella] Support Flexible
Auto Queue Creation in Capacity Scheduler
[ https://issues.apache.org/jira/browse/YARN-10496?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17242282#comment-17242282 ]
Peter Bacsko edited comment on YARN-10496 at 12/2/20, 11:42 AM:
----------------------------------------------------------------
My take: clearly I think it's between #1 and #2.
Both require re-calculating the underlying percentages, but in case of #1 it's hidden from us because our fundamental unit is weight. However, current CS users might be perfectly happy with the renormalization to 100 so they might opt for #2.
But since this issue came up in the context of FS-to-CS migration, it's probably more important to FS users. So I'd vote for approach #1, introducing weight mode.
was (Author: pbacsko):
My take: clearly I think it's between #1 and #2.
Both require re-calculating the underlying percentages, but in case of #1 it's hidden from us because our fundamental unit is weight. However, current CS users might be perfectly happy with the renormalization to 100 so they might opt for #2.
But since this issue came up during FS-to-CS migration, it's probably more important to FS users. So I'd vote for approach #1, introducing weight mode.
> [Umbrella] Support Flexible Auto Queue Creation in Capacity Scheduler
> ---------------------------------------------------------------------
>
> Key: YARN-10496
> URL: https://issues.apache.org/jira/browse/YARN-10496
> Project: Hadoop YARN
> Issue Type: New Feature
> Components: capacity scheduler
> Reporter: Wangda Tan
> Priority: Major
>
> CapacityScheduler today doesn’t support an auto queue creation which is flexible enough. The current constraints:
> * Only leaf queues can be auto-created
> * A parent can only have either static queues or dynamic ones. This causes multiple constraints. For example:
> * It isn’t possible to have a VIP user like Alice with a static queue root.user.alice with 50% capacity while the other user queues (under root.user) are created dynamically and they share the remaining 50% of resources.
>
> * In comparison, FairScheduler allows the following scenarios, Capacity Scheduler doesn’t:
> ** This implies that there is no possibility to have both dynamically created and static queues at the same time under root
> * A new queue needs to be created under an existing parent, while the parent already has static queues
> * Nested queue mapping policy, like in the following example:
> |<rule name="nestedUserQueue" create=”true”>
> <rule name="primaryGroup" create="true" />
> </rule>|
> * Here two levels of queues may need to be created
> If an application belongs to user _alice_ (who has the primary_group of _engineering_), the scheduler checks whether _root.engineering_ exists, if it doesn’t, it’ll be created. Then scheduler checks whether _root.engineering.alice_ exists, and creates it if it doesn't.
>
> When we try to move users from FairScheduler to CapacityScheduler, we face feature gaps which blocks users migrate from FS to CS.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org