You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Eric Payne (Jira)" <ji...@apache.org> on 2020/09/11 20:03:00 UTC
[jira] [Commented] (YARN-950) Ability to limit or avoid aggregating
logs beyond a certain size
[ https://issues.apache.org/jira/browse/YARN-950?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17194507#comment-17194507 ]
Eric Payne commented on YARN-950:
---------------------------------
[~adam.antal], are you still planning on working on this JIRA?
bq. Ran into another case where a user filled a disk with a large stdout/stderr, and the NM took forever to recover the disk
bq. +1 on the idea of limiting the size of log aggregation. We should have someway to truncate (front & tail) the log contents if it is really large.
It seems that there are a couple of requirements for this feature.
# Prevent logs for any container from becoming larger than a configurable size.
# Truncate everything from the middle of large log files, leaving a configurable amount of the head and the tail.
Internally, we have implemented the former by allowing system admins to specify a configurable max log size per container.
> Ability to limit or avoid aggregating logs beyond a certain size
> ----------------------------------------------------------------
>
> Key: YARN-950
> URL: https://issues.apache.org/jira/browse/YARN-950
> Project: Hadoop YARN
> Issue Type: Sub-task
> Components: log-aggregation, nodemanager
> Affects Versions: 0.23.9, 2.6.0
> Reporter: Jason Darrell Lowe
> Assignee: Adam Antal
> Priority: Major
>
> It would be nice if ops could configure a cluster such that any container log beyond a configured size would either only have a portion of the log aggregated or not aggregated at all. This would help speed up the recovery path for cases where a container creates an enormous log and fills a disk, as currently it tries to aggregate the entire, enormous log rather than only aggregating a small portion or simply deleting it.
--
This message was sent by Atlassian Jira
(v8.3.4#803005)
---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org