[jira] [Commented] (YARN-8199) Logging fileSize of log files under NM Local Dir

["Adam Antal (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/YARN-8199?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16885150#comment-16885150 ] 

Adam Antal commented on YARN-8199:
----------------------------------

Thanks for patch [~Prabhu Joseph],

1) When one of the {{getFileStatus}} call fails all of the other files are not getting processed since the catch block is out of the loop. That would not be a problem for HDFS, but if another filesystem is given as remote-app-folder where {{getFileStatus}} is not reliable then it can produce unexpected messages (S3!). Also it would make this debugging not reliable at all: if a file is not mentioned there in the logs, we could not deduce that this log has actually not exceeded the limit given with the configuration.
2) I agree putting this to a debug block, since the {{getFileStatus}} reasoning above. However I'd recommend mentioning this in the description: it only takes effect if level at least debug (even though the configuration itself implicitly contains "debug").
3) Could please add the container id (and maybe the application id as well) when displaying the message about the size of the file? If one app has lots of containers some of those having syslog exceeding the size then the error message can not be bound to a container.
4) I'd rename yarn.log-aggregation.debug.filesize.bytes to yarn.log-aggregation.debug.filesize since you already mention in the description the measure.

Just out of curiosity: how did you end up with 100Mb as the limit?

> Logging fileSize of log files under NM Local Dir
> ------------------------------------------------
>
>                 Key: YARN-8199
>                 URL: https://issues.apache.org/jira/browse/YARN-8199
>             Project: Hadoop YARN
>          Issue Type: Improvement
>          Components: log-aggregation
>    Affects Versions: 2.7.3
>            Reporter: Prabhu Joseph
>            Assignee: Prabhu Joseph
>            Priority: Major
>              Labels: supportability
>         Attachments: 0001-YARN-8199.patch, 0002-YARN-8199.patch, YARN-8199-003.patch
>
>
> Logging fileSize of log files like syslog, stderr, stdout under NM Local Dir by NodeManager before the cleanup will help to find the application which has written too verbose.



--
This message was sent by Atlassian JIRA
(v7.6.14#76016)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org