[jira] [Commented] (YARN-2176) CapacityScheduler loops over all running applications rather than actively requesting apps

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2176?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14514506#comment-14514506 ] 

Wangda Tan commented on YARN-2176:
----------------------------------

[~jlowe], this problem should be resolved in CS after YARN-3361, in LeafQueue:
{code}
      if (!application.hasPendingResourceRequest(resourceCalculator,
          node.getPartition(), clusterResource, schedulingMode)) {
        if (LOG.isDebugEnabled()) {
          LOG.debug("Skip app_attempt=" + application.getApplicationAttemptId()
              + ", because it doesn't need more resource, schedulingMode="
              + schedulingMode.name() + " node-label=" + node.getPartition());
        }
        continue;
      }
{code}
To only look into apps has pending requests.

> CapacityScheduler loops over all running applications rather than actively requesting apps
> ------------------------------------------------------------------------------------------
>
>                 Key: YARN-2176
>                 URL: https://issues.apache.org/jira/browse/YARN-2176
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: capacityscheduler
>    Affects Versions: 2.4.0
>            Reporter: Jason Lowe
>
> The capacity scheduler performance is primarily dominated by LeafQueue.assignContainers, and that currently loops over all applications that are running in the queue.  It would be more efficient if we looped over just the applications that are actively asking for resources rather than all applications, as there could be thousands of applications running but only a few hundred that are currently asking for resources.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)