[jira] [Commented] (HDFS-2794) HA: Active NN may purge edit log files before standby NN has a chance to read them

["Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-2794?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201082#comment-13201082 ] 

Todd Lipcon commented on HDFS-2794:
-----------------------------------

Attached patch adds a new configuration, dfs.namenode.num.extra.edits.retained, which causes the NN to not purge a given number of edits that are older than the oldest retained local checkpoint. This seemed preferable to me than the other option discussed, which was to configure the NN to retain many more images. The reason is that even a million edits (the default) would be on the order of a few hundred MB, whereas retaining a day's worth of checkpoints might be on the order of hundreds of GB for a large cluster making frequent checkpoints.

Retaining edits for a long period of time has some other useful applications, as well (eg a binary form of audit log).
                
> HA: Active NN may purge edit log files before standby NN has a chance to read them
> ----------------------------------------------------------------------------------
>
>                 Key: HDFS-2794
>                 URL: https://issues.apache.org/jira/browse/HDFS-2794
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Aaron T. Myers
>            Assignee: Todd Lipcon
>         Attachments: hdfs-2794.txt
>
>
> Given that the active NN is solely responsible for purging finalized edit log segments, and given that the active NN has no way of knowing when the standby reads edit logs, it's  possible that the standby NN could fail to read all edits it needs before the active purges the files.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira