[jira] [Updated] (YARN-4751) In 2.7, Labeled queue usage not shown properly in capacity scheduler UI

["Eric Payne (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-4751?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Eric Payne updated YARN-4751:
-----------------------------
    Attachment: YARN-4751-branch-2.7.003.patch

Thank you, [~leftnoteasy] and [~sunilg], for your reviews and helpful insights.

I backported YARN-3362 to 2.7, but it did not provide all of the correct statistics for labeled queues all by itself.
- Used Capacity was not calculated correctly for node labels because {{QueueCapacities#setUsedCapacity}} does not have a label-specific interface that could set the capacity per partition. In order to get the updated changes to {{CSQueueUtils#updateQueueStatistics}} that set used capacity per label, I would need to backport YARN-2696. I did not do that since it also had other dependencies and was more invasive than I felt was needed.
-- Instead, I added a {{getUsedCapacity}} method to {{AbstractCSQueue}} that takes a label and returns the computed used capacity for the queue for that label.
- The {{RMNodeLabel}} class is named {{NodeLabel}} in 2.7.
- The {{CapacitySchedulerInfo}} constructor doesn't have the {{CapacityScheduler}} parameter in 2.7, so I left it out (as you suggested previously).
- The YARN-3362 patch has a dependency on {{AbstractCSQueue#accessibleToPartition}} which was not in 2.7. Rather than backport YARN-3361 as well, I just recrated {{accessibleToPartition}} in this patch (as you suggested previously).
- One of the changes for {{CapacitySchedulerInfo#getQueues}} in YARN-3362 added a check to skip non-accessible queues. In 2.7, there are 2 loops that iterate over the queues whereas in 2.8, there is only 1. I put the check in the first loop because the first loop is filtering the queues to be processed by the second loop 
- I modified {{AbstractCSQueue#allocateResource}} and {{releaseResource}} to intersect with all labels in the passed {{nodeLabels}} set if the accessible node labels included the {{ANY}} selection. Without this, queue usage for root would not be preserved per-partiton and the root queue's bar graph would not display any usage at all for specific partitions.
- Modified {{LeafQueue#getUser}} to sum all labeled and unlabeld used resources for a user on a queue. Without this, the {{Used Resource}} in the {{Active Users Info}} section would be 0.


> In 2.7, Labeled queue usage not shown properly in capacity scheduler UI
> -----------------------------------------------------------------------
>
>                 Key: YARN-4751
>                 URL: https://issues.apache.org/jira/browse/YARN-4751
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: capacity scheduler, yarn
>    Affects Versions: 2.7.3
>            Reporter: Eric Payne
>            Assignee: Eric Payne
>         Attachments: 2.7 CS UI No BarGraph.jpg, YARH-4752-branch-2.7.001.patch, YARH-4752-branch-2.7.002.patch, YARN-4751-branch-2.7.003.patch
>
>
> In 2.6 and 2.7, the capacity scheduler UI does not have the queue graphs separated by partition. When applications are running on a labeled queue, no color is shown in the bar graph, and several of the "Used" metrics are zero.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)