You are viewing a plain text version of this content. The canonical link for it is here.
Posted to yarn-issues@hadoop.apache.org by "Junping Du (JIRA)" <ji...@apache.org> on 2017/08/28 20:49:00 UTC

[jira] [Comment Edited] (YARN-7083) Log aggregation deletes/renames while file is open

    [ https://issues.apache.org/jira/browse/YARN-7083?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16144317#comment-16144317 ] 

Junping Du edited comment on YARN-7083 at 8/28/17 8:48 PM:
-----------------------------------------------------------

Trunk code has changed significantly since YARN-6876 so patch here doesn't apply any more. From my quick look, the issue (not close file before rename/delete) is still there but the fix is not straightforward as writer get hidden behind for different formats. I would suggest to commit the patch fix here to branch-2.8 and branch-2.8.2 only and create a separated jira tracking for trunk/branch-2. 
[~jlowe], what do you think?


was (Author: djp):
Trunk code has changed significantly since YARN-6877 so patch here doesn't apply any more. From my quick look, the issue (not close file before rename/delete) is still there but the fix is not straightforward as writer get hidden behind for different formats. I would suggest to commit the patch fix here to branch-2.8 and branch-2.8.2 only and create a separated jira tracking for trunk/branch-2. 
[~jlowe], what do you think?

> Log aggregation deletes/renames while file is open
> --------------------------------------------------
>
>                 Key: YARN-7083
>                 URL: https://issues.apache.org/jira/browse/YARN-7083
>             Project: Hadoop YARN
>          Issue Type: Bug
>          Components: nodemanager
>    Affects Versions: 2.8.2
>            Reporter: Daryn Sharp
>            Assignee: Jason Lowe
>            Priority: Critical
>         Attachments: YARN-7083.001.patch
>
>
> YARN-6288 changes the log aggregation writer to be an autoclosable.  Unfortunately the try-with-resources block for the writer will either rename or delete the log while open.
> Assuming the NM's behavior is correct, deleting open files only results in ominous WARNs in the nodemanager log and increases the rate of logging in the NN when the implicit try-with-resource close fails.  These red herrings complicate debugging efforts.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)

---------------------------------------------------------------------
To unsubscribe, e-mail: yarn-issues-unsubscribe@hadoop.apache.org
For additional commands, e-mail: yarn-issues-help@hadoop.apache.org