[jira] [Created] (YARN-10821) User limit is not calculated as per definition for preemption

["Andras Gyori (Jira)" <ji...@apache.org>]

Andras Gyori created YARN-10821:
-----------------------------------

             Summary: User limit is not calculated as per definition for preemption
                 Key: YARN-10821
                 URL: https://issues.apache.org/jira/browse/YARN-10821
             Project: Hadoop YARN
          Issue Type: Bug
          Components: capacity scheduler
            Reporter: Andras Gyori
            Assignee: Andras Gyori


Minimum user limit percent (MULP) is a soft limit by definition. Preemption uses pending resources to determine the resources needed by a queue, which is calculated in LeafQueue#getTotalPendingResourcesConsideringUserLimit. This method involves headroom calculated by UsersManager#computeUserLimit. However, the pending resources for preemption are limited in an unexpected fashion.
 * In LeafQueue#getUserAMResourceLimitPerPartition an effective userLimit is calculated first:
{code:java}
 float effectiveUserLimit = Math.max(usersManager.getUserLimit() / 100.0f,
 1.0f / Math.max(getAbstractUsersManager().getNumActiveUsers(), 1));
{code}

 * In UsersManager#computeUserLimit the userLimit is calculated as is (currentCapacity * userLimit)
{code:java}
 Resource userLimitResource = Resources.max(resourceCalculator,
 partitionResource,
 Resources.divideAndCeil(resourceCalculator, resourceUsed,
 usersSummedByWeight),
 Resources.divideAndCeil(resourceCalculator,
 Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
 100));
{code}

The fewer users occupying the queue, the more prevalent and outstanding this effect will be in preemption.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org