[jira] [Created] (YARN-11022) Fix the documentation for max-parallel-apps in CS

["Tamas Domok (Jira)" <ji...@apache.org>]

Tamas Domok created YARN-11022:
----------------------------------

             Summary: Fix the documentation for max-parallel-apps in CS
                 Key: YARN-11022
                 URL: https://issues.apache.org/jira/browse/YARN-11022
             Project: Hadoop YARN
          Issue Type: Bug
          Components: capacity scheduler
    Affects Versions: 3.4.0
            Reporter: Tamas Domok
            Assignee: Tamas Domok


The documentation does not mention that the max-parallel-apps property is inherited. The property can be overridden on a per queue basis, but the parent(s) can also restrict how many parallel apps can be run.

 

{*}yarn.scheduler.capacity.max-parallel-apps / yarn.scheduler.capacity.<queue-path>.max-parallel-apps{*}: Maximum number of applications that can run at the same time. Unlike to maximum-applications, application submissions are not rejected when this limit is reached. Instead they stay in ACCEPTED state until they are eligible to run. This can be set for all queues with yarn.scheduler.capacity.max-parallel-apps and can also be overridden on a per queue basis by setting yarn.scheduler.capacity.<queue-path>.max-parallel-apps. Integer value is expected. By default, there is no limit.

 

[https://github.com/apache/hadoop/blob/03cfc852791c14fad39db4e5b14104a276c08e59/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CSMaxRunningAppsEnforcer.java#L99]
  private boolean exceedQueueMaxParallelApps(AbstractCSQueue queue) {
    // Check queue and all parent queues    while (queue != null) {
      if (queue.getNumRunnableApps() >= queue.getMaxParallelApps()) {
        LOG.info("Maximum runnable apps exceeded for queue {}",
            queue.getQueuePath());
        return true;
      }
      queue = (AbstractCSQueue) queue.getParent();
    }    return false;
  } 

Example:



Let's say the user configured the *yarn.scheduler.capacity.max-parallel-apps* to 250, that will be the default for queues that doesn't override the setting. ([https://github.com/apache/hadoop/blob/32ecaed9c3c06a48ef01d0437e62e8faccd3e9f3/hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager/src/main/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacitySchedulerConfiguration.java#L1688])

Given this queue hierarchy:
||root||.a||.a1||.a2||.a3||.a4||
|500|default|50|10|default|15|
||root||.a||.b||
|500|default|50|
 - maximum 250 apps can run parallel under the *root.a* queues.
 - maximum 50 apps can run parallel under the *root.a.a1* queues.
 - maximum 10 apps can run parallel under the *root.a.a1.a2* queues.
 - maximum *10* apps can run parallel under the *root.a1.a2.a3* queues. (even though the max-parallel-apps is not set for .a3 so the default 250 applies for that queue, but it's parent had a lower value, and children can't exceed that)
 - maximum *10* apps can run parallel under the *root.a1.a2.a3.a4* queue. (even though it's configured for 15, the parents restrict this limit to 10)
 - maximum 50 apps can run parallel under the *root.a.b* queue.



--
This message was sent by Atlassian Jira
(v8.20.1#820001)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org