You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Eli Collins (Created) (JIRA)" <ji...@apache.org> on 2012/01/05 22:16:39 UTC
[jira] [Created] (HDFS-2752) HA: exit if multiple shared dirs are
configured
HA: exit if multiple shared dirs are configured
-----------------------------------------------
Key: HDFS-2752
URL: https://issues.apache.org/jira/browse/HDFS-2752
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha
Affects Versions: HA branch (HDFS-1623)
Reporter: Eli Collins
Assignee: Eli Collins
We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180859#comment-13180859 ]
Eli Collins commented on HDFS-2752:
-----------------------------------
It's not currently, but it should be until we support it (see HDFS-2735, needs tests and fixes).
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180853#comment-13180853 ]
Suresh Srinivas commented on HDFS-2752:
---------------------------------------
Why is this an error?
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Suresh Srinivas (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180894#comment-13180894 ]
Suresh Srinivas commented on HDFS-2752:
---------------------------------------
Eli, I am confused a bit. What is the special required in supporting multiple shared directories that is not handled currently? Why turn it off because tests are not there?
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Jitendra Nath Pandey (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201552#comment-13201552 ]
Jitendra Nath Pandey commented on HDFS-2752:
--------------------------------------------
bq. I agree with Eli - we don't currently use the JournalSet abstraction in EditLogTailer, so it can only use a single shared dir. Of course in the future we should support using multiple, but it adds some complexity to the initial release.
EditLogTailer uses FSEditLog which uses JournalSet. I think it should be able to handle multiple shared edits, unless there is another bug.
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201568#comment-13201568 ]
Todd Lipcon commented on HDFS-2752:
-----------------------------------
The other issue is that, in order to support multiple shared edits, we'd need a quorum-like behavior rather than the current "at least one" behavior. Otherwise you could imagine that, with two shared dirs (SD1 and SD2) and two NNs, you might have the case where NN1 is writing to only SD1 and NN2 is reading from only SD2. Let's continue this discussion on HDFS-2782
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13183052#comment-13183052 ]
Todd Lipcon commented on HDFS-2752:
-----------------------------------
I agree with Eli - we don't currently use the JournalSet abstraction in EditLogTailer, so it can only use a single shared dir. Of course in the future we should support using multiple, but it adds some complexity to the initial release.
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Hudson (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201259#comment-13201259 ]
Hudson commented on HDFS-2752:
------------------------------
Integrated in Hadoop-Hdfs-HAbranch-build #70 (See [https://builds.apache.org/job/Hadoop-Hdfs-HAbranch-build/70/])
HDFS-2752. HA: exit if multiple shared dirs are configured. Contributed by Eli Collins
eli : http://svn.apache.org/viewcvs.cgi/?root=Apache-SVN&view=rev&rev=1240916
Files :
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/CHANGES.HDFS-1623.txt
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/main/java/org/apache/hadoop/hdfs/server/namenode/FSNamesystem.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/MiniDFSCluster.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/TestFSNamesystem.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestFailureOfSharedDir.java
* /hadoop/common/branches/HDFS-1623/hadoop-hdfs-project/hadoop-hdfs/src/test/java/org/apache/hadoop/hdfs/server/namenode/ha/TestHAConfiguration.java
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Eli Collins (Updated) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins updated HDFS-2752:
------------------------------
Attachment: hdfs-2752.txt
Patch attached. Running the full test suite for sanity. I left the dupe detection for shared dirs in place since we'll need it when we add multiple shared dir support.
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Todd Lipcon (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13201091#comment-13201091 ]
Todd Lipcon commented on HDFS-2752:
-----------------------------------
Just one nit: you can remove the following empty javadoc annotation:
{code}
+ * @param conf
{code}
Aside from that, +1.
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Eli Collins (Commented) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13180953#comment-13180953 ]
Eli Collins commented on HDFS-2752:
-----------------------------------
Sorry for the lack of context, this came up in HDFS-2709. With multiple shared edits dirs a failure to read from one of them will prevent the edit log tailer from catching up, ie users are currently *less reliable* with multiple shared dirs. Until we know we've got it working reasonably well (ie have tests for the common scenarios) it doesn't seem like we should let people shoot themselves in the foot. And while nice to have, it doesn't seem like multiple shared dir support should block an initial release. Agree? Perhaps warn loudly instead of exit?
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-2752) HA: exit if multiple shared dirs are
configured
Posted by "Eli Collins (Resolved) (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-2752?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Eli Collins resolved HDFS-2752.
-------------------------------
Resolution: Fixed
Hadoop Flags: Reviewed
Thanks for the review Todd. Fixed the nit and committed.
> HA: exit if multiple shared dirs are configured
> -----------------------------------------------
>
> Key: HDFS-2752
> URL: https://issues.apache.org/jira/browse/HDFS-2752
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha
> Affects Versions: HA branch (HDFS-1623)
> Reporter: Eli Collins
> Assignee: Eli Collins
> Attachments: hdfs-2752.txt
>
>
> We don't support multiple shared edits dirs, we should fail to start with an error in this case.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira