[jira] [Resolved] (YARN-6947) The implementation of Schedulable#getResourceUsage so inefficiency that can reduce the performance of scheduling

["Yufei Gu (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/YARN-6947?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Yufei Gu resolved YARN-6947.
----------------------------
    Resolution: Duplicate

> The implementation of Schedulable#getResourceUsage so inefficiency that can reduce the performance of scheduling
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-6947
>                 URL: https://issues.apache.org/jira/browse/YARN-6947
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: YunFan Zhou
>            Priority: Critical
>
> Each time the FairScheduler assign container, it will checks whether the resources used by the queue exceed Max Share. However, our current calculation of the resources of the queue is particularly inefficient, which recursively iterates over all child nodes, with high time complexity. 
> We can refactor this logic by using lazy update way.
> {code:java}
>   @Override
>   public Resource assignContainer(FSSchedulerNode node) {
>     Resource assigned = Resources.none();
>     // If this queue is over its limit, reject
>     if (!assignContainerPreCheck(node)) {
>       return assigned;
>     }
> {code}
> {code:java}
>    * Helper method to check if the queue should attempt assigning resources
>    * 
>    * @return true if check passes (can assign) or false otherwise
>    */
>   boolean assignContainerPreCheck(FSSchedulerNode node) {
>     if (node.getReservedContainer() != null) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because it has reserved containers.");
>       }
>       return false;
>     } else if (!Resources.fitsIn(getResourceUsage(), maxShare)) {
>       if (LOG.isDebugEnabled()) {
>         LOG.debug("Assigning container failed on node '" + node.getNodeName()
>             + " because queue resource usage is larger than MaxShare: "
>             + dumpState());
>       }
>       return false;
>     } else {
>       return true;
>     }
>   }
> {code}
> {code:java}
>   @Override
>   public Resource getResourceUsage() {
>     Resource usage = Resources.createResource(0);
>     readLock.lock();
>     try {
>       for (FSQueue child : childQueues) {
>         Resources.addTo(usage, child.getResourceUsage());
>       }
>     } finally {
>       readLock.unlock();
>     }
>     return usage;
>   }
> {code}



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org