[jira] [Created] (YARN-8364) NM aggregation thread should be able to exempt pool

["Oleksandr Shevchenko (JIRA)" <ji...@apache.org>]

Oleksandr Shevchenko created YARN-8364:
------------------------------------------

             Summary: NM aggregation thread should be able to exempt pool
                 Key: YARN-8364
                 URL: https://issues.apache.org/jira/browse/YARN-8364
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: log-aggregation
            Reporter: Oleksandr Shevchenko


For now, we have limited NM aggregation thread pool that can be configured by the property yarn.nodemanager.logaggregation.threadpool-size-max=100. 
When some application is starting it use one unit of the pool. And locks this unit until the application is finished. As the result, another application can aggregate their logs only when the previous application is finished.

Just for example:
yarn.nodemanager.logaggregation.threadpool-size-max=1

1. Start long-running application app1
2. Start short application app2
3. Finished app2
4. Finished app1
5. Aggregating logs of app1
6. Aggregating logs of app2

In the real cluster, we can have many long running jobs (for example Spark streaming), therefore short-running application do not aggregate their logs a long time. It problem appears if the average number of jobs exceeds thread pool size. All threads occupied by some applications, as the result we have the huge delay between application finishing and logs uploading.

Will be good if we improve this behavior.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-dev-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-dev-help@hadoop.apache.org