You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Payne (Jira)" <ji...@apache.org> on 2021/07/06 22:01:00 UTC
[jira] [Comment Edited] (YARN-10821) User limit is not calculated
as per definition for preemption
[ https://issues.apache.org/jira/browse/YARN-10821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17376054#comment-17376054 ]
Eric Payne edited comment on YARN-10821 at 7/6/21, 10:00 PM:
-------------------------------------------------------------
{quote}
{noformat}
max ((total resources consumed by active users) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
bq. UserA is not using any resource yet, and no one else is using QueueA. Is not "resource usage by active users" expected to be low in this case?
{quote}
So, you know how I said that the above was a simplification? Here is another wrinkle in that algorithm:
IF queueUsage <= queueCapacity
THEN
(total resources consumed by active users) = yarn.scheduler.capacity.root.QueueA.capacity
ENDIF
So, basically, in the use case where QueueA is empty, the algorithm becomes:
{noformat}
max ((yarn.scheduler.capacity.root.QueueA.capacity) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
was (Author: eepayne):
{quote}
{noformat}
max ((total resources consumed by active users) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
bq. UserA is not using any resource yet, and no one else is using QueueA. Is not "resource usage by active users" expected to be low in this case?
{quote}
So, you know how I said that the above was a simplification? Here is another wrinkle in that algorithm:
{noformat}
IF queueUsage <= queueCapacity
THEN
(total resources consumed by active users) = yarn.scheduler.capacity.root.QueueA.capacity
ENDIF
So, basically, in the use case where QueueA is empty, the algorithm becomes:
{noformat}
max ((yarn.scheduler.capacity.root.QueueA.capacity) / (number of active users)),
((totalQueueResourcesAvailableToUsers) * min-user-limit-pct))
{noformat}
> User limit is not calculated as per definition for preemption
> -------------------------------------------------------------
>
> Key: YARN-10821
> URL: https://issues.apache.org/jira/browse/YARN-10821
> Project: Hadoop YARN
> Issue Type: Bug
> Components: capacity scheduler
> Reporter: Andras Gyori
> Assignee: Andras Gyori
> Priority: Major
> Attachments: YARN-10821.001.patch
>
>
> Minimum user limit percent (MULP) is a soft limit by definition. Preemption uses pending resources to determine the resources needed by a queue, which is calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This method involves headroom calculated by UsersManager#computeUserLimit. However, the pending resources for preemption are limited in an unexpected fashion.
> * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is calculated first:
> {code:java}
> float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
> 1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
> {code}
> * In UsersManager#computeUserLimit the userLimit is calculated as is (currentCapacity * userLimit)
> {code:java}
> Resource userLimitResource = Resources.max(resourceCalculator,
> partitionResource,
> Resources.divideAndCeil(resourceCalculator, resourceUsed,
> usersSummedByWeight),
> Resources.divideAndCeil(resourceCalculator,
> Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
> 100));
> {code}
> The fewer users occupying the queue, the more prevalent and outstanding this effect will be in preemption.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org