You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Sunil Govindan (JIRA)" <ji...@apache.org> on 2018/11/17 03:24:11 UTC
[jira] [Updated] (YARN-4953) Delete completed container log folder
when rolling log aggregation is enabled
[ https://issues.apache.org/jira/browse/YARN-4953?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Sunil Govindan updated YARN-4953:
---------------------------------
Target Version/s: 3.3.0 (was: 3.2.0)
Bulk update: moved all 3.2.0 non-blocker issues, please move back if it is a blocker.
> Delete completed container log folder when rolling log aggregation is enabled
> -----------------------------------------------------------------------------
>
> Key: YARN-4953
> URL: https://issues.apache.org/jira/browse/YARN-4953
> Project: Hadoop YARN
> Issue Type: Bug
> Components: nodemanager
> Reporter: Rohith Sharma K S
> Assignee: Rohith Sharma K S
> Priority: Major
>
> There would be potential bottle neck when cluster is running with very large number of containers on the same NodeManager for single application. The linux limits the subfolders count to 32K. If number of containers is greater than 32K for an application, there would be container launch failure. At this point of time, there are no more containers can be launched in this node.
> Currently log folders are deleted after app is finished. Rolling log aggregation aggregates logs to hdfs periodically.
> I think if aggregation is completed for finished containers, then clean up can be done i.e deleting log folder for finished containers.
--
This message was sent by Atlassian JIRA
(v7.6.3#76005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org