You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Nathan Roberts (JIRA)" <ji...@apache.org> on 2012/10/17 01:09:03 UTC
[jira] [Created] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Nathan Roberts created YARN-162:
-----------------------------------
Summary: nodemanager log aggregation has scaling issues with namenode
Key: YARN-162
URL: https://issues.apache.org/jira/browse/YARN-162
Project: Hadoop YARN
Issue Type: Bug
Components: nodemanager
Affects Versions: 0.23.3
Reporter: Nathan Roberts
Priority: Critical
Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500447#comment-13500447 ]
Hadoop QA commented on YARN-162:
--------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12554200/YARN-162_v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/160//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/160//console
This message is automatically generated.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_v2.txt, YARN-162_v2.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500286#comment-13500286 ]
Hadoop QA commented on YARN-162:
--------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12553856/YARN-162.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/155//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/155//console
This message is automatically generated.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499191#comment-13499191 ]
Robert Joseph Evans commented on YARN-162:
------------------------------------------
The patch looks good as an initial start, but I would have to dig into it more to really understand what is happening.
In doAppLogAggregation after creating the log dir, if that fails we shut down the file system and send an event saying the app finished but we go on trying to aggregate the logs anyways. Also we are going to close the FileSystem later on no matter what anyways too so it is probably unnecessary to close it here.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Attachments: YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500300#comment-13500300 ]
Robert Joseph Evans commented on YARN-162:
------------------------------------------
Sid I like the patch. I have a few minor comments
# there are a few TODOs added into the code. {code}// TODO This is broken. Container ID for the AM may not be 1.{code}, {code}// TODO Should the app really fail if log aggregation fails ?{code} and {code}// TODO Send out an event to the app. Currently since aggregation failure{code}. I could not find an existing JIRA for the first one so please file one for it. The other two seem to be related to one another. If you feel strongly that we should not fail an application because log aggregation will not work, then please file a separate JIRA for that, otherwise the TODOs should just be comments and not TODOs.
# I don't really like the name of the new config that was added. It exposes the internal implementation of how we throttle the applications. I would prefer to have it called something like yarn.nodemanager.log-aggregation.max-concurrent-apps. But this is very minor.
# The new config was not added to yarn-default.xml
# This is also very minor. Inside LogAggregationService.stopApp we are wrapping a Void callable inside another Void callable. I would prefer it if we returned the original value instead of returning null.
With Jenkin's +1 I am OK with the change, but it is a large enough change that I am a bit nervous about pulling this into 0.23.5. If you are OK with this, I will pull in a modified YARN-219 that addresses your comments, and then we can pull this into trunk, branch-2, and branch-0.23 (0.23.6)
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13478098#comment-13478098 ]
Vinod Kumar Vavilapalli commented on YARN-162:
----------------------------------------------
Pity we didn't foresee this.
The NM has per-app file handles open, but it doesn't close them till the application finishes.
Arguably, this connections are to the data-nodes, but the DFSClient also keeps a transient connection to NNs which seem to be causing this.
Will investigate more and take a stab at fixing this.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Posted by "Robert Joseph Evans (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Joseph Evans updated YARN-162:
-------------------------------------
Assignee: Siddharth Seth (was: Vinod Kumar Vavilapalli)
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated YARN-162:
--------------------------------
Attachment: YARN-162_WIP.txt
Work in progress patch for YARN-219, YARN-220.
Is a reasonably simple change - the aggregator thread for an app is started after the app completes, the aggregator takes care of directory creation and there's a limited number of threads for aggregation.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Attachments: YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499108#comment-13499108 ]
Siddharth Seth commented on YARN-162:
-------------------------------------
Bobby, if you haven't gotten started working on these yet , I'm half way through a patch for YARN-219 and YARN-220.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Assigned] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Vinod Kumar Vavilapalli (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Vinod Kumar Vavilapalli reassigned YARN-162:
--------------------------------------------
Assignee: Vinod Kumar Vavilapalli
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13499232#comment-13499232 ]
Siddharth Seth commented on YARN-162:
-------------------------------------
The patch does change behaviour. Prior to this, directory creation failure would cause the app to be marked as FAILED on the NM. Failing to aggregate logs for some containers would not fail the app though. With the patch, dir creation failing does not fail the app.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13505654#comment-13505654 ]
Hadoop QA commented on YARN-162:
--------------------------------
{color:green}+1 overall{color}. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12554200/YARN-162_v2.txt
against trunk revision .
{color:green}+1 @author{color}. The patch does not contain any @author tags.
{color:green}+1 tests included{color}. The patch appears to include 1 new or modified test files.
{color:green}+1 javac{color}. The applied patch does not increase the total number of javac compiler warnings.
{color:green}+1 javadoc{color}. The javadoc tool did not generate any warning messages.
{color:green}+1 eclipse:eclipse{color}. The patch built with eclipse:eclipse.
{color:green}+1 findbugs{color}. The patch does not introduce any new Findbugs (version 1.3.9) warnings.
{color:green}+1 release audit{color}. The applied patch does not increase the total number of release audit warnings.
{color:green}+1 core tests{color}. The patch passed unit tests in hadoop-yarn-project/hadoop-yarn/hadoop-yarn-common hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-nodemanager.
{color:green}+1 contrib tests{color}. The patch passed contrib unit tests.
Test results: https://builds.apache.org/job/PreCommit-YARN-Build/180//testReport/
Console output: https://builds.apache.org/job/PreCommit-YARN-Build/180//console
This message is automatically generated.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_v2.txt, YARN-162_v2.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated YARN-162:
--------------------------------
Attachment: YARN-162.txt
Updated patch. Fixes the additional filesystem close, and a bunch of other problems with the last one.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Vinod Kumar Vavilapalli
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated YARN-162:
--------------------------------
Attachment: YARN-162_v2.txt
Updated post YARN-219.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_v2.txt, YARN-162_v2.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (YARN-162) nodemanager log aggregation has
scaling issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13500352#comment-13500352 ]
Siddharth Seth commented on YARN-162:
-------------------------------------
Thanks for the review.
bq. With Jenkin's +1 I am OK with the change, but it is a large enough change that I am a bit nervous about pulling this into 0.23.5. If you are OK with this, I will pull in a modified YARN-219 that addresses your comments, and then we can pull this into trunk, branch-2, and branch-0.23 (0.23.6)
Fair enough. I'll address the review comments and post another patch.
bq. The other two seem to be related to one another. If you feel strongly that we should not fail an application because log aggregation will not work, then please file a separate JIRA for that, otherwise the TODOs should just be comments and not TODOs.
Without this patch, I believe log aggregation will ignore errors in aggregating logs for individual containers. It'll pass as long as the app directory can be created. The patch changes things to allow dir creation to fail as well. If a user asks for log-aggregation, and any part of it fails - should the app fail ? IAC, will create another jira.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (YARN-162) nodemanager log aggregation has scaling
issues with namenode
Posted by "Siddharth Seth (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/YARN-162?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Siddharth Seth updated YARN-162:
--------------------------------
Attachment: YARN-162_v2.txt
Patch addressing review comments.
> nodemanager log aggregation has scaling issues with namenode
> ------------------------------------------------------------
>
> Key: YARN-162
> URL: https://issues.apache.org/jira/browse/YARN-162
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Affects Versions: 0.23.3
> Reporter: Nathan Roberts
> Assignee: Siddharth Seth
> Priority: Critical
> Attachments: YARN-162.txt, YARN-162_v2.txt, YARN-162_WIP.txt
>
>
> Log aggregation causes fd explosion on the namenode. On large clusters this can exhaust FDs to the point where datanodes can't check-in.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira