[jira] [Commented] (HDFS-2874) HA: edit log should log to shared dirs before local dirs

["Aaron T. Myers (Commented) (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/HDFS-2874?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13198450#comment-13198450 ] 

Aaron T. Myers commented on HDFS-2874:
--------------------------------------

It occurs to me that using the "required" dirs as a proxy for the shared dirs isn't necessarily correct. It's true that the shared dir should always be marked required, but an admin might also configure some local or other dirs to be required. Were they to do this, the sync order would again be undefined, and the shared dir might not get logged first, and thus we might hit this case.

So, it seems like we either need to push down the shared/non-shared distinction into the JournalSet, or we need to somehow push down a generic "sync order" concept into the journal set. Or, perhaps, guarantee that the journalset syncs to journals in the order given at initialization.
                
> HA: edit log should log to shared dirs before local dirs
> --------------------------------------------------------
>
>                 Key: HDFS-2874
>                 URL: https://issues.apache.org/jira/browse/HDFS-2874
>             Project: Hadoop HDFS
>          Issue Type: Sub-task
>          Components: ha, name-node
>    Affects Versions: HA branch (HDFS-1623)
>            Reporter: Todd Lipcon
>            Assignee: Todd Lipcon
>            Priority: Critical
>         Attachments: hdfs-2874.txt
>
>
> Currently, the NN logs its edits to each of its edits directories in sequence. This can produce the following bad sequence:
> - NN accumulates 100 edits (tx 1-100) in the buffer. Writes and syncs to local drive, then crashes
> - Failover occurs. SBN takes over at txid=1, since txid 1 never got writen.
> - First NN restarts. It reads up to txid 100 from its local directories. It is now "ahead" of the active NN with inconsistent state.
> The solution is to write to the shared edits dir, and sync that, before writing to any local drives.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira