[jira] [Commented] (YARN-6788) Improve performance of resource profile branch

["Wangda Tan (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-6788?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16080868#comment-16080868 ] 

Wangda Tan commented on YARN-6788:
----------------------------------

Thanks [~sunilg], comments:

Resource:
- {{public abstract Integer getResourceIndex(String resource)}} should be internal API, how about only add it to {{ResourceUtils.getResourceTypeIndex(String resourceName)}}. We should try to avoid store any local information of resource_type_to_index.

ResourcePBImpl:
- What does the readOnlyResources means?

bq. In Resource class, there are few impls for SimpleResource. I also had an offline talk with Varun Vasudev. May be its better to move all resource related impl from ResourcePBImpl to Resource class itself and can keep only proto related changes in PBImpl class. So SimpleResource could look into memory and cpu resource info objects.
I suggest to not take this shortcut, we have several built-in resource types are adding to YARN, such as GPU/FPGA, I don't want to get unexpected performance regression after adding them.

Resources/DominantResourceCalculator (Maybe there're more places could be changed)
Now they're using either {{setResourceValue(name, value)}}, or {{getResourceInformation(rName)}}. Both of them will do frequent map-looking operations. Instead of doing this, can we add a public (marked as {{@private}}) APIs to Resource object, which support get/setResourceInformation/Value with index. Internally we can use it to do computations.
To me all name-related fields should not be used while doing computations. String-based names should be only used for human-readability, such as UI/message, etc. 

(Not performance related), there're many places in the code hardcoded to use {{set/getVirtualCores}} and assume there're only 2 resource types. I suggest to review all of them before merge the branch. 

> Improve performance of resource profile branch
> ----------------------------------------------
>
>                 Key: YARN-6788
>                 URL: https://issues.apache.org/jira/browse/YARN-6788
>             Project: Hadoop YARN
>          Issue Type: Sub-task
>          Components: nodemanager, resourcemanager
>            Reporter: Sunil G
>            Assignee: Sunil G
>            Priority: Blocker
>         Attachments: YARN-6788-YARN-3926.001.patch
>
>
> Currently we could see a 15% performance delta with this branch. 
> Few performance improvements to improve the same.
> Also this patch will handle [comments|https://issues.apache.org/jira/browse/YARN-6761?focusedCommentId=16075418&page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel#comment-16075418] from [~leftnoteasy].



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org