You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (Created) (JIRA)" <ji...@apache.org> on 2012/01/12 00:21:50 UTC
[jira] [Created] (HDFS-2782) HA: Support multiple shared edits dirs
HA: Support multiple shared edits dirs
--------------------------------------
Key: HDFS-2782
URL: https://issues.apache.org/jira/browse/HDFS-2782
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Eli Collins
Assignee: Eli Collins
Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2782) HA: Support multiple shared edits
dirs
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201583#comment-13201583 ]
Todd Lipcon commented on HDFS-2782:
-----------------------------------
Been thinking about this a bit... I don't think it's so trivial as to just use our existing JournalSet implementation across multiple remote directories. The issue is the following case:
- HA cluster configured with two NNs: NN1 and NN2, and two shared storage directories (SD1 and SD2)
- NN1 is active and writing to SD1 and SD2 when a network issue occurs which partitions NN1 from SD2 and NN2 from SD1 (this isn't that unlikely, for example if NN1 and SD1 share a rack while NN2 and SD2 share a rack).
- If a failover occurs at this point, NN2 could take over without reading the most recent edits from SD1, resulting in a divergent namespace.
I have two possible solutions in mind:
*Solution 1: use a traditional quorum system*
When a NN writes, require that all writes must be synced to _W_ shared storage directories. When the SBN reads, require that it read edits from _R_ shared storage directories. The quorum requirement is that _R + W > N_ where _N_ is the total number of shared dirs.
In the error case described above, we could set W = 1 and R = 2. Thus the active NN could continue to operate even if one of the SDs is down -- but no failovers could occur during that window of time. Once the directory is restored, the system would be fully operational again and ready for failover.
The usual quorum requirement that W > N/2 would not be necessary here, since we already ensure a designed single writer by fencing operations.
*Solution 2: use an external source of record to agree on storage directory state.*
In this solution, an external system (likely zookeeper) is used to agree upon the state of the storage directories. ZK would contain a znode which lists the active shared directories. When the active NN writes, it writes to all of these active directories. If any of the writes fail, it must update the znode to mark the failed directory as out-of-date before acking the write.
When a directory is restored, it will be re-added to the znode listing active directories.
When the SBN processes a failover, it considers only those directories listed in the znode as active.
Any other solutions I'm not thinking of here?
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2782) HA: Support multiple shared edits dirs
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2782:
---------------------------------
Issue Type: New Feature (was: Sub-task)
Reporter: Aaron T. Myers (was: Eli Collins)
Parent: (was: HDFS-1623)
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2782) HA: Support multiple shared edits
dirs
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201897#comment-13201897 ]
Eli Collins commented on HDFS-2782:
-----------------------------------
There's also a variant of solution #2 where you don't use an external source, eg NN1 and NN2 communicate directly to determine a common set of shared dirs (they re-negotiate when either sees a SD fail). In your partition scenario failover would not be possible because NN2 can not access all the directories it previously negotiated with NN1. This isn't a good solution though because it requires both the active and standby be available whenever a SD fails (and the list needs to be re-negotiated).
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2782) HA: Support multiple shared edits dirs
Posted by "Aaron T. Myers (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-2782:
---------------------------------
Target Version/s: 0.24.0 (was: HA branch (HDFS-1623))
Affects Version/s: (was: HA branch (HDFS-1623))
0.24.0
Converting to top-level issue with commit of HDFS-1623.
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2782) HA: Support multiple shared edits
dirs
Posted by "Sanjay Radia (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13185359#comment-13185359 ]
Sanjay Radia commented on HDFS-2782:
------------------------------------
2 questions:
- If you have multiple dirs within the same NFS server, are you expecting them to fail independently?
- For multiple NFS servers, will the standby be expected to read from any/all of them?
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2782) HA: Support multiple shared edits
dirs
Posted by "Eli Collins (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins resolved HDFS-2782.
-------------------------------
Resolution: Won't Fix
Assignee: (was: Eli Collins)
Target Version/s: (was: 0.24.0)
Given QJM (HDFS-3077) IMO this is no longer worth considering.
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: ha
> Affects Versions: 0.24.0
> Reporter: Aaron T. Myers
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2782) HA: Support multiple shared edits
dirs
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2782?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13186039#comment-13186039 ]
Eli Collins commented on HDFS-2782:
-----------------------------------
#1 Expecting both independent failures (eg the dirs are one two separate mounts with different underlying volumes may fail independently) and dependent failures (if the connection to the same server is severed for example).
#2 Yes, a 2nd or 3rd shared dir is just like the first. Eg if the primary or standby marked a shared dir as bad it and the standby could switch to another.
> HA: Support multiple shared edits dirs
> --------------------------------------
>
> Key: HDFS-2782
> URL: https://issues.apache.org/jira/browse/HDFS-2782
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> Supporting multiple shared dirs will improve availability (eg see HDFS-2769). You may want to use multiple shared dirs on a single filer (eg for better fault isolation) or because you want to use multiple filers/mounts. Per HDFS-2752 (and HDFS-2735) we need to do things like use the JournalSet in EditLogTailer and add tests.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira