[jira] [Updated] (YARN-7473) Implement Framework and policy for capacity management of auto created queues

["Suma Shivaprasad (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-7473?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Suma Shivaprasad updated YARN-7473:
-----------------------------------
    Attachment: YARN-7473.11.patch

Thanks [~sunilg] and [~wangda] Attaching patch with most of the review comments addressed 

{quote} Once queue capacity is set to 0, same queue will be assigned back with some capacity based on available capacity. If capacity is not there, queue will be still 0. If many such queues with 0 capacity is starving, queue which got submitted with an app first will be selected. We might need to consider priority also here. {quote}
   Discussed with [~wangda] and [~sunilg] offline. App priority is intra queue and will not be usable as a criteria while considering to choose leaf queues 
{quote} AbstractManagedParentQueue#validateQueueEntitlementChange is directly operating on capacity. When absolute resource will get merged, this code will be a problem? {quote} 
   This will require changes in policy class and could be addressed in a separate jira once Aboslute resource is merged.
In initializeLimitsFromTemplate, should below code to be setMaxApplications(leafQueueTemplate.getMaxApps());
related to parent queue as well. What if some one provided more apps in template which could violate parent max-apps? 
{quote} In validateConfigurations, does 0 a valid capacity? one could configure 0 as capacity and +ve integer for max-capacity?{quote}
Yes, 0 is a valid configuration for queue capacity 

{quote}
How to configure orderingPolicy for AutoCreatedLeafQueue?
in below code, better to avoid _ for queue config names
{quote}
Can be configured through <parent-queue-prefix>.leaf-queue-template.ordering-policy.

{quote}
1684	  public static final String QUEUE_MANAGEMENT_MONITORING_INTERVAL =
1685	      QUEUE_MANAGEMENT_CONFIG_PREFIX + "monitoring_interval";
{quote} 
   Fixed
{quote}
In CapacitySchedulerContext, better to use MonotonicClock
{quote}
  Fixed

{quote}
In LeafQueue,
2011	  public void setMaxAMResourcePerQueuePercent(
2012	      float maxAMResourcePerQueuePercent) {
2013	    this.maxAMResourcePerQueuePercent = maxAMResourcePerQueuePercent;
2014	  }
how are we handling node labels?
{quote} 
  Discussed with [~wangda] and [~sunilg] offline. This has been addressed through changes for using CapacitySchedulerConfiguration instead of individual configs for a ManagedParentQueue

{quote} 
PendingApplicationComparator could reuse existing fifo/fair app comparators?
{quote} 
  Discussed with [~wangda] and [~sunilg] offline. fifo/fair app comparators cant be reused since they are mostly for intra queue app ordering and we cannot consider priority etc while ordering leaf queues within a parent queue as mentioned above.


Also this patch reverts changes done for using a common AutoCreatedLeafQueue for both reservations and auto created queues. This was needed since the initialize/reinitialize for ReservationQueues and AutoCreatedLeafQueue are no longer common.

> Implement Framework and policy for capacity management of auto created queues 
> ------------------------------------------------------------------------------
>
>                 Key: YARN-7473
>                 URL: https://issues.apache.org/jira/browse/YARN-7473
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: capacity scheduler
>            Reporter: Suma Shivaprasad
>            Assignee: Suma Shivaprasad
>         Attachments: YARN-7473.1.patch, YARN-7473.10.patch, YARN-7473.11.patch, YARN-7473.2.patch, YARN-7473.3.patch, YARN-7473.4.patch, YARN-7473.5.patch, YARN-7473.6.patch, YARN-7473.7.patch, YARN-7473.8.patch, YARN-7473.9.patch
>
>
> This jira mainly addresses the following
>  
> 1.Support adding pluggable policies on parent queue for dynamically managing capacity/state for leaf queues.
> 2. Implement  a default policy that manages capacity based on pending applications and either grants guaranteed or zero capacity to queues based on parent's available guaranteed capacity.
> 3. Integrate with SchedulingEditPolicy framework to trigger this periodically and signal scheduler to take necessary actions for capacity/queue management.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org