[jira] [Created] (YARN-6491) Move totalMB and totalVirtualCores computation from ClusterMetricsInfo to QueueMetrics

["Jonathan Hung (JIRA)" <ji...@apache.org>]

Jonathan Hung created YARN-6491:
-----------------------------------

             Summary: Move totalMB and totalVirtualCores computation from ClusterMetricsInfo to QueueMetrics
                 Key: YARN-6491
                 URL: https://issues.apache.org/jira/browse/YARN-6491
             Project: Hadoop YARN
          Issue Type: Improvement
            Reporter: Jonathan Hung


Right now in ClusterMetricsInfo.java we do this:{noformat}    if (rs instanceof CapacityScheduler) {
      this.totalMB = metrics.getTotalMB();
      this.totalVirtualCores = metrics.getTotalVirtualCores();
    } else {
      this.totalMB = availableMB + allocatedMB;
      this.totalVirtualCores = availableVirtualCores + allocatedVirtualCores;
    }{noformat}

We'd like to have totalMB and totalVirtualCores as QueueMetrics fields. But since QueueMetrics is scheduler agnostic we can't really just move this. It seems the way totalMB and totalVirtualCores is computed across FS and CS should be standardized.

Right now CS does not include reservedMB in allocatedMB, while FS does (as far as I can tell). At least in <= 2.7, when a container is reserved, queueUsage.getUsed is incremented, which is the value used to determine if a queue can be assigned (AbstractCSQueue#canAssignToThisQueue). So I think it makes sense to increment allocatedMB when a container is reserved, and not increment it when a reserved container is allocated, to reflect the fact that if allocated + reserved > queueLimit, allocation will fail, so the allocatedMB metric should also be > queueLimit.

Still not sure if the same is true >= 2.8. Would appreciate any input on this (or any of the mentioned proposals).



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org