[jira] [Commented] (HDFS-3092) Enable journal protocol based editlog streaming for standby namenode

["Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-3092?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13244341#comment-13244341 ] 

Todd Lipcon commented on HDFS-3092:
-----------------------------------

Hi Suresh. I took a look at the design document, and I think it actually shares a lot with what I'm doing in HDFS-3077. Hopefully we can share some portions of the code and design.

Here are some points I think need elaboration in the design doc:
- How does the fencing command ensure that prior NNs can no longer access the JD after it completes? I think you need to have the JDs record the sequence number of each NN so they can reject past NNs from coming back to life.
- I don't think the following can be done correctly:
{quote}
a. Choose the transaction >= Q JDs from JournalList.
b. Else choose the highest transaction ID from a JD in the JournalList.
3. All the JDs perform recovery to transaction ID Ft .
{quote}
because it's possible that there are distinct transactions with the same txid at the beginning of a log segment. For example, consider the following situation with two NN (NN1 and NN2) and three JDs (JD1, JD2, JD3):

1. NN1 writes txid 1 to JD1, crashes before writing to JD2 and JD3 
2. NN2 initiates fencing, but only succeeds in contacting JD2 and JD3. So, it does not see the edit made in step 1
3. NN2 writes txid1 to JD2 and JD3, then crashes
4. One of the two NNs recovers. It sees that all JDs have txid 1. The fencing/synchronization process you've described cannot distinguish between the correct txn which was written to a quorum and the incorrect txn which was only written to JD1.
                
> Enable journal protocol based editlog streaming for standby namenode
> --------------------------------------------------------------------
>
>                 Key: HDFS-3092
>                 URL: https://issues.apache.org/jira/browse/HDFS-3092
>             Project: Hadoop HDFS
>          Issue Type: Improvement
>          Components: ha, name-node
>    Affects Versions: 0.24.0, 0.23.3
>            Reporter: Suresh Srinivas
>            Assignee: Suresh Srinivas
>         Attachments: MultipleSharedJournals.pdf
>
>
> Currently standby namenode relies on reading shared editlogs to stay current with the active namenode, for namespace changes. BackupNode used streaming edits from active namenode for doing the same. This jira is to explore using journal protocol based editlog streams for the standby namenode. A daemon in standby will get the editlogs from the active and write it to local edits. To begin with, the existing standby mechanism of reading from a file, will continue to be used, instead of from shared edits, from the local edits.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira