[jira] Commented: (HDFS-1073) Simpler model for Namenode's fs Image and edit Logs

["Ivan Kelly (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-1073?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12915783#action_12915783 ] 

Ivan Kelly commented on HDFS-1073:
----------------------------------

{quote}
    while writing edit logs to multiple files, a failure of the th system can result in different amounts of data written to each file - the tid allows one to pick one with the most tranasactions.

Isn't this also doable by just seeing which as more non-zero bytes? ie seek to the end of the file, scan backwards through the 0 bytes, and stop. Whichever valid log is longer wins. Even in the case with the transaction-id, you have to do something like this for a few reasons: a) we'd rather scan backward from the end of the edit log than forward from the beginning, since it's going to be a faster startup, and b) even if we see a higher transaction id header on the last entry, that entry might have been incompletely written to the file, so we still have to verify that it deserializes correctly.
{quote}

The case of all edit logs being _inprogress during a crash should be a very rare case. Is it really an issue if it takes a little longer to determine which has the most transactions if it's only going to incurred after a bad crash?

{quote}
I don't think either way has been decided/rejected yet. What you're saying has been my view - that doing txid based is a bigger change, since we have to introduce the txid concept and add extra code that allows replaying partial edit log files (ie a subrange of the edits within). But it's certainly doable and Sanjay has presented some good advantages.
{quote}

FSEditLog already has a transaction concept which could be modified for this. Currently its not stored anywhere, but is used for logSync. It starts and 0 at NN startup and increases monotonically, reseting the next time NN starts.



> Simpler model for Namenode's fs Image and edit Logs 
> ----------------------------------------------------
>
>                 Key: HDFS-1073
>                 URL: https://issues.apache.org/jira/browse/HDFS-1073
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>            Reporter: Sanjay Radia
>            Assignee: Todd Lipcon
>         Attachments: hdfs1073.pdf
>
>
> The naming and handling of  NN's fsImage and edit logs can be significantly improved resulting simpler and more robust code.

-- 
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.