[jira] [Commented] (YARN-2768) optimize FSAppAttempt.updateDemand by avoid clone of Resource which takes 85% of computing time of update thread

["Hong Zhiguo (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-2768?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14581349#comment-14581349 ] 

Hong Zhiguo commented on YARN-2768:
-----------------------------------

[~kasha], the excution time displayed in the profiling output is cumulative.
Actually, I repeated such profiling a lot of times and got the same ratio.
The profiling is done with a cluster of NM/AM simulators and I don't have such resource now.

I wrote a testcase which creates 8000 nodes, 4500 apps within 1200 queues, and then performs 10000 rounds of FairScheduler.update(), and print the average execution time of one call to update. With this patch, the average execution time decreased from about 35ms to 20ms.

I think the effect comes from GC and memory allocation since in each round of FairScheduler.update(), Resource.multiply is called as many times as the number of pending ResourceRequests, which is more than 3 million in our production cluster.

> optimize FSAppAttempt.updateDemand by avoid clone of Resource which takes 85% of computing time of update thread
> ----------------------------------------------------------------------------------------------------------------
>
>                 Key: YARN-2768
>                 URL: https://issues.apache.org/jira/browse/YARN-2768
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: fairscheduler
>            Reporter: Hong Zhiguo
>            Assignee: Hong Zhiguo
>            Priority: Minor
>         Attachments: YARN-2768.patch, profiling_FairScheduler_update.png
>
>
> See the attached picture of profiling result. The clone of Resource object within Resources.multiply() takes up **85%** (19.2 / 22.6) CPU time of the function FairScheduler.update().
> The code of FSAppAttempt.updateDemand:
> {code}
> public void updateDemand() {
>     demand = Resources.createResource(0);
>     // Demand is current consumption plus outstanding requests
>     Resources.addTo(demand, app.getCurrentConsumption());
>     // Add up outstanding resource requests
>     synchronized (app) {
>       for (Priority p : app.getPriorities()) {
>         for (ResourceRequest r : app.getResourceRequests(p).values()) {
>           Resource total = Resources.multiply(r.getCapability(), r.getNumContainers());
>           Resources.addTo(demand, total);
>         }
>       }
>     }
>   }
> {code}
> The code of Resources.multiply:
> {code}
> public static Resource multiply(Resource lhs, double by) {
>     return multiplyTo(clone(lhs), by);
> }
> {code}
> The clone could be skipped by directly update the value of this.demand.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)