You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Payne (JIRA)" <ji...@apache.org> on 2018/08/10 18:13:00 UTC
[jira] [Comment Edited] (YARN-8509) Total pending resource
calculation in preemption should use user-limit factor instead of
minimum-user-limit-percent
[ https://issues.apache.org/jira/browse/YARN-8509?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16576342#comment-16576342 ]
Eric Payne edited comment on YARN-8509 at 8/10/18 6:12 PM:
-----------------------------------------------------------
[~Zian Chen], can I please get a couple of clarifications?
{quote}
total_pending(partition,queue) = min {Q_max(partition) - Q_used(partition), Σ (min \{
User.ulf\(partition\) - User.used\(partition\), User.pending\(partition\}\)\}
{quote}
1) In the above pseudo-code, what is being summed by the summation?
2) In the above example, queue-a is the only one that's underserved, so the the first round of preemption should actually preempt 6G from queues b and c. The amount preempted from each queue depends on the age of the containers, but you could end up with something like queue-b consuming 40G and pending 30G and queue-c consuming 44G and pending 36G before the second round of preemption, at which point queue-a would be satisfied and only queues b and c have pending resource requests. Since this issue is meant to address the balancing of queues that are over their capacity, I don't understand why queue-a is involved in the above use case. Can you provide a simpler example that only involves the balancing of over-served queues?
was (Author: eepayne):
[~Zian Chen], can I please get a couple of clarifications?
{quote}total_pending(partition,queue) = min {Q_max(partition) - Q_used(partition), Σ (min
Unknown macro: \{User.ulf(partition) - User.used(partition), User.pending(partition})}{quote}
1) In the above pseudo-code, what is being summed by the summation?
2) In the above example, queue-a is the only one that's underserved, so the the first round of preemption should actually preempt 6G from queues b and c. The amount preempted from each queue depends on the age of the containers, but you could end up with something like queue-b consuming 40G and pending 30G and queue-c consuming 44G and pending 36G before the second round of preemption, at which point queue-a would be satisfied and only queues b and c have pending resource requests. Since this issue is meant to address the balancing of queues that are over their capacity, I don't understand why queue-a is involved in the above use case. Can you provide a simpler example that only involves the balancing of over-served queues?
> Total pending resource calculation in preemption should use user-limit factor instead of minimum-user-limit-percent
> -------------------------------------------------------------------------------------------------------------------
>
> Key: YARN-8509
> URL: https://issues.apache.org/jira/browse/YARN-8509
> Project: Hadoop YARN
> Issue Type: Bug
> Components: yarn
> Reporter: Zian Chen
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8509.001.patch, YARN-8509.002.patch, YARN-8509.003.patch
>
>
> In LeafQueue#getTotalPendingResourcesConsideringUserLimit, we calculate total pending resource based on user-limit percent and user-limit factor which will cap pending resource for each user to the minimum of user-limit pending and actual pending. This will prevent queue from taking more pending resource to achieve queue balance after all queue satisfied with its ideal allocation.
>
> We need to change the logic to let queue pending can go beyond userlimit.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org