[jira] [Created] (YARN-9055) Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration

["Aihua Xu (JIRA)" <ji...@apache.org>]

Aihua Xu created YARN-9055:
------------------------------

             Summary: Capacity Scheduler: allow larger queue level maximum-allocation-mb to override the cluster configuration
                 Key: YARN-9055
                 URL: https://issues.apache.org/jira/browse/YARN-9055
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: capacityscheduler
    Affects Versions: 2.7.0
            Reporter: Aihua Xu
            Assignee: Aihua Xu


YARN-1582 adds the support of maximum-allocation-mb configuration per queue. That feature gives the flexibility to give different memory requirements for different queues. Such patch adds the limitation that the queue level configuration can't exceed the cluster level default configuration, but I feel it may make more sense to remove such limitation to allow any overrides since 
# Such configuration is controlled by the admin so it shouldn't get abused; 
# It's common that typical queues require standard size containers while some job (queues) have requirements for larger containers. With current limitation, we have to set larger configuration on the cluster setting which will cause resource abuse unless we override them on all the queues.

We can remove such limitation in CapacitySchedulerConfiguration.java so the cluster setting provides the default value and queue setting can override it. 

{noformat}
   if (maxAllocationMbPerQueue > clusterMax.getMemorySize()
        || maxAllocationVcoresPerQueue > clusterMax.getVirtualCores()) {
      throw new IllegalArgumentException(
          "Queue maximum allocation cannot be larger than the cluster setting"
          + " for queue " + queue
          + " max allocation per queue: " + result
          + " cluster setting: " + clusterMax);
    }
{noformat}

Let me know if it makes sense.







--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org