[jira] [Commented] (YARN-11414) ClusterMetricsInfo shows wrong availableMB when node labels enabled

["ASF GitHub Bot (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-11414?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17739689#comment-17739689 ] 

ASF GitHub Bot commented on YARN-11414:
---------------------------------------

tomicooler commented on PR #5293:
URL: https://github.com/apache/hadoop/pull/5293#issuecomment-1618880984

   Hi,
   
   I can help in reviewing the code, but I'm not familiar with ClusterMetrics or FairScheduler.
   
   ```
   2. Currently, ClusterMetricsInfo shows available or allocated memory only for Default Partition. I don't think this is by intention, if so there has to be at least one test case failed with this patch. From the name ClusterMetricsInfo, it is Cluster Wide, showing only for Default Partition does not look right and also does not help clusters where some/all nodes are part of non-default partitions.
   ```
   Agreed. I also think, that we should add a new test case that validates the behaviour.
   
   E.g.: in the `TestRMWebServicesCapacitySched`:
   ```
     @Test
     public void testClusterMetrics() throws Exception {
       try (MockRM rm = createRM(createConfigWithNodeLabels())){
         rm.registerNode("h1:1234", 32 * GB, 32); // multiple nodes with some labels that makes sense
         assertJsonResponse(resource().path("ws/v1/cluster/metrics")
                 .accept(MediaType.APPLICATION_JSON).get(ClientResponse.class),
             "webapp/scheduler-response.json");
       }
     }
   ```
   
   
   ```
   3. Without this patch, currently i see there is no way to retrieve the Cluster Wide Available and Allocated Memory. But we have ways to get it for Partition level already through QueueMetrics . Having both ClusterMetrics and QueueMetrics showing for Default Partition seems redundant.
   ```
   
   I think the `availableMB` and `allocatedMB` can be calculated from the `ClusterMetrics` for CapacityScheduler without this change, e.g.:
   ```
     availableMB = totalClusterResourcesAcrossPartition .getMemorySize() - totalUsedResourcesAcrossPartition.getMemorySize()
     allocatedMB = totalUsedResourcesAcrossPartition.getMemorySize()
   ```




> ClusterMetricsInfo shows wrong availableMB when node labels enabled 
> --------------------------------------------------------------------
>
>                 Key: YARN-11414
>                 URL: https://issues.apache.org/jira/browse/YARN-11414
>             Project: Hadoop YARN
>          Issue Type: Bug
>    Affects Versions: 3.3.3
>            Reporter: Prabhu Joseph
>            Assignee: Ashutosh Gupta
>            Priority: Major
>              Labels: pull-request-available
>
> ClusterMetricsInfo shows wrong availableMB when node labels enabled. It shows availableMB of Default Partition alone. 



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org