You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Zian Chen (JIRA)" <ji...@apache.org> on 2018/06/06 02:24:00 UTC
[jira] [Commented] (YARN-8379) Add an option to allow Capacity
Scheduler preemption to balance satisfied queues
[ https://issues.apache.org/jira/browse/YARN-8379?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16502751#comment-16502751 ]
Zian Chen commented on YARN-8379:
---------------------------------
Thanks, [~leftnoteasy] for bringing up this feature. I worked on this Jira and finish an initial patch for the first stage of this feature which enables fairness balance after all queues beyond its guarantee. What I did is,
# Release the constraint in TempQueuePerPartition#offer to let queues can continue to grows its ideal assigned resource after it satisfied it's guaranteed.
# Make fairness balance preemption can be configurable through capacity-scheduler.xml
# Add several test cases in PCPP to make sure this part of logic works for DRF and defaultResourceCalculator.
What I plan to do in the second stage is,
# Add timeout to candidateSelector and make fairness balance selector has its different timeout.
# Figure out the list of containers each candidateSelector selected since we need this information to set the different timeout for selected containers from fairness balance selector
Currently, in each step of candidate selection, selector always make changes in the same "toPreempt" set of RMContainer and only stores aggregated results, besides this information, we also need individual results from each selector as well.
[~eepayne] , could you help review the patch for the first stage and see if there is any potential issue for changing the tempQueuePerPartition part? Also, could you share how you accomplished the fairness feature in 2.8 as you mentioned? Thanks!
> Add an option to allow Capacity Scheduler preemption to balance satisfied queues
> --------------------------------------------------------------------------------
>
> Key: YARN-8379
> URL: https://issues.apache.org/jira/browse/YARN-8379
> Project: Hadoop YARN
> Issue Type: Bug
> Reporter: Wangda Tan
> Assignee: Zian Chen
> Priority: Major
> Attachments: YARN-8379.001.patch
>
>
> Existing capacity scheduler only supports preemption for an underutilized queue to reach its guaranteed resource. In addition to that, there’s an requirement to get better balance between queues when all of them reach guaranteed resource but with different fairness resource.
> An example is, 3 queues with capacity, queue_a = 30%, queue_b = 30%, queue_c = 40%. At time T. queue_a is using 30%, queue_b is using 70%. Existing scheduler preemption won't happen. But this is unfair to queue_a since queue_a has the same guaranteed resources.
> Before YARN-5864, capacity scheduler do additional preemption to balance queues. We changed the logic since it could preempt too many containers between queues when all queues are satisfied.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org