[jira] [Commented] (YARN-10009) DRF can treat minimum user limit percent as a max when custom resource is defined

["Eric Payne (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10009?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16986411#comment-16986411 ] 

Eric Payne commented on YARN-10009:
-----------------------------------

The root cause is here:
{code:UsersManager#computeUserLimit}
    /*
     * User limit resource is determined by: max(currentCapacity / #activeUsers,
     * currentCapacity * user-limit-percentage%)
     */
    Resource userLimitResource = Resources.max(resourceCalculator,
        partitionResource,
        Resources.divideAndCeil(resourceCalculator, resourceUsed,
            usersSummedByWeight),
        Resources.divideAndCeil(resourceCalculator,
            Resources.multiplyAndRoundDown(currentCapacity, getUserLimit()),
            100));
{code}
When calculating the user resource limit, {{divideAndCeil}} is used to take the max of either (queue capacity / # of active users) or (queue capacity / min user limit pct). However, they are not the same divideAndCeil methods. The first takes a {{Resource}} and a {{float}} and the second takes a {{Resource}} and an {{int}}. The method with the {{Resource}} {{float}} signature was never updated to handle custom resources.

The only place that calls {{difideAndCeil(Resource, float)}} is here in {{UsersManager#computeUserLimit}}

> DRF can treat minimum user limit percent as a max when custom resource is defined
> ---------------------------------------------------------------------------------
>
>                 Key: YARN-10009
>                 URL: https://issues.apache.org/jira/browse/YARN-10009
>             Project: Hadoop YARN
>          Issue Type: Improvement
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>            Priority: Major
>
> | | Memory | Vcores | res_1 |
> | Queue1 Totals | 20GB | 100 | 80 |
> | Resources requested by App1 in Queue1 | 8GB (40% of total) | 8 (8% of total) | 80 (100% of total) |
> In the previous use case:
> - Queue1 has a value of 25 for {{miminum-user-limit-percent}}
> - User1 has requested 8 containers with {{<memory:1GB, vcores:8, res_1:10>}} each
> - {{res_1}} will be the dominant resource this case.
> All 8 containers should be assigned by the capacity scheduler, but with min user limit pct set to 25, only 3 containers are assigned.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org