You are viewing a plain text version of this content. The canonical link for it is here.

Posted to yarn-dev@hadoop.apache.org by "Haibo Chen (JIRA)" <ji...@apache.org> on 2016/02/17 01:06:18 UTC

[jira] [Created] (YARN-4697) NM aggregation thread pool is not bound by limits

Haibo Chen created YARN-4697:
--------------------------------

             Summary: NM aggregation thread pool is not bound by limits
                 Key: YARN-4697
                 URL: https://issues.apache.org/jira/browse/YARN-4697
             Project: Hadoop YARN
          Issue Type: Improvement
          Components: nodemanager
            Reporter: Haibo Chen
            Assignee: Haibo Chen


In the LogAggregationService.java we create a threadpool to upload logs from the nodemanager to HDFS if log aggregation is turned on. This is a cached threadpool which based on the javadoc is an ulimited pool of threads.
In the case that we have had a problem with log aggregation this could cause a problem on restart. The number of threads created at that point could be huge and will put a large load on the NameNode and in worse case could even bring it down due to file descriptor issues.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)