[jira] [Commented] (YARN-10821) User limit is not calculated as per definition for preemption

["Eric Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10821?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17490455#comment-17490455 ] 

Eric Payne commented on YARN-10821:
-----------------------------------

[~gandras], do you still believe we need this JIRA? My opinion is that user limit calculations _are_ correct for preemption.

> User limit is not calculated as per definition for preemption
> -------------------------------------------------------------
>
>                 Key: YARN-10821
>                 URL: https://issues.apache.org/jira/browse/YARN-10821
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler
>            Reporter: Andras Gyori
>            Assignee: Andras Gyori
>            Priority: Major
>         Attachments: YARN-10821.001.patch
>
>
> Minimum user limit percent (MULP) is a soft limit by definition. Preemption uses pending resources to determine the resources needed by a queue, which is calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This method involves headroom calculated by UsersManager#computeUserLimit. However, the pending resources for preemption are limited in an unexpected fashion.
>  * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is calculated first:
> {code:java}
>  float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
>  1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
> {code}
>  * In UsersManager#computeUserLimit the userLimit is calculated as is (currentCapacity * userLimit)
> {code:java}
>  Resource userLimitResource = Resources.max(resourceCalculator,
>  partitionResource,
>  Resources.divideAndCeil(resourceCalculator, resourceUsed,
>  usersSummedByWeight),
>  Resources.divideAndCeil(resourceCalculator,
>  Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
>  100));
> {code}
> The fewer users occupying the queue, the more prevalent and outstanding this effect will be in preemption.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org