[jira] [Commented] (YARN-10504) Implement weight mode in Capacity Scheduler

["Andras Gyori (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-10504?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17259772#comment-17259772 ] 

Andras Gyori commented on YARN-10504:
-------------------------------------

Fixed the following test classes:
 * TestCSMaxRunningAppsEnforcer
 * TestNodeLabelContainerAllocation
 * TestQueueParsing
 * TestCapacitySchedulerDynamicBehavior
 * TestCapacitySchedulerPlanFollower

I have identified a problematic part of the change, which must be addressed, because it affects a lot of classes. 
The *CSQueueUtils* has been changed and now only the *loadCapacitiesByLabelsFromConf* method is used. Several tests are failing because:
{code:java}
float absCapacity = queueCapacities.getCapacity(label);
float absMaxCapacity = queueCapacities.getMaximumCapacity(label);
if (absCapacity > absMaxCapacity) {
  throw new IllegalArgumentException("Illegal queue capacity setting "
      + "(abs-capacity=" + absCapacity + ") > (abs-maximum-capacity="
      + absMaxCapacity + ") for queue=["
      + queuePath + "],label=[" + label + "]");
}{code}
Now, it might be a typo or a bug, because absCapacity here is NOT the actual absolute capacity, but the regular capacity. This leads to an other problem, namely the absolute capacity is not calculated for LeafQueue and PlanQueue, which leaves the absolute capacity as 0. This was previously calculated in *CSQueueUtils*. The *AbstractCSQueue*#*updateAbsoluteCapacity* contains the same logic, but the AutoCreatedLeafQueue tests are still failing, and probably due to this issue.
The questions are:
 * Is the check in loadCapacitiesByLabelsFromConf correct, or absCapacity should be equals to queueCapacities.getAbsolutCapacity()
 * Where do we calculate the absolute capacity, which is not handled now in two queue classes?

 

> Implement weight mode in Capacity Scheduler
> -------------------------------------------
>
>                 Key: YARN-10504
>                 URL: https://issues.apache.org/jira/browse/YARN-10504
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>            Reporter: Benjamin Teke
>            Assignee: Benjamin Teke
>            Priority: Major
>         Attachments: YARN-10504.001.patch, YARN-10504.002.patch, YARN-10504.ver-1.patch, YARN-10504.ver-2.patch, YARN-10504.ver-3.patch
>
>
> To allow the possibility to flexibly create queues in Capacity Scheduler a weight mode should be introduced. The existing \{{capacity }}property should be used with a different syntax, i.e:
> root.users.capacity = (1.0) or ~1.0 or ^1.0 or @1.0
> root.users.capacity = 1.0w
> root.users.capacity = w:1.0
> Weight support should not impact the existing functionality.
>  
> The new functionality should: 
>  * accept and validate the new weight values
>  * enforce a singular mode on the whole queue tree
>  * (re)calculate the relative (percentage-based) capacities based on the weights during launch and every time the queue structure changes



--
This message was sent by Atlassian Jira
(v8.3.4#803005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org