You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Aaron T. Myers (JIRA)" <ji...@apache.org> on 2012/09/12 07:39:07 UTC
[jira] [Created] (HDFS-3926) QJM: Add user documentation for QJM
Aaron T. Myers created HDFS-3926:
------------------------------------
Summary: QJM: Add user documentation for QJM
Key: HDFS-3926
URL: https://issues.apache.org/jira/browse/HDFS-3926
Project: Hadoop HDFS
Issue Type: Sub-task
Components: documentation
Affects Versions: QuorumJournalManager (HDFS-3077)
Reporter: Aaron T. Myers
Assignee: Aaron T. Myers
We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3926:
---------------------------------
Attachment: regular-ha-doc.diff
qjm-ha-doc.diff
HDFS-3926.patch
Thanks a lot for the comments, Todd. Here's a patch which addresses all of them as you suggested, except for the following:
bq. This paragraph no longer seems relevant.
Agree that part of it is now irrelevant with the QJM, but not the whole thing. I took out the part of this paragraph that said the fencing was critical, but left in the part that describes how to configure the fencing methods.
This patch also takes the liberty of correcting the value for the dfs.ha.fencing.ssh.connect-timeout sample configuration, to no longer be malformed XML.
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-3926.patch, HDFS-3926.patch, qjm-ha-doc.diff, qjm-ha-doc.diff, regular-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Resolved] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers resolved HDFS-3926.
----------------------------------
Resolution: Fixed
Fix Version/s: QuorumJournalManager (HDFS-3077)
Hadoop Flags: Reviewed
I've just committed this to the QJM branch.
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Fix For: QuorumJournalManager (HDFS-3077)
>
> Attachments: HDFS-3926.patch, HDFS-3926.patch, HDFS-3926.patch, qjm-ha-doc.diff, qjm-ha-doc.diff, regular-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3926:
---------------------------------
Attachment: HDFS-3926.patch
Thanks a lot, Todd. Here's an updated patch which fixes those two little issues.
I'm going to commit this momentarily.
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-3926.patch, HDFS-3926.patch, HDFS-3926.patch, qjm-ha-doc.diff, qjm-ha-doc.diff, regular-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13454201#comment-13454201 ]
Todd Lipcon commented on HDFS-3926:
-----------------------------------
> order to actually increase the number of failures the system can tolerate, you
> should run an odd number of JNs, (i.e. 3, 5, 7, etc.)
Probably should explicitly say that, with N JNs, we tolerate (N - 1)/2 failures.
> * <<dfs.namenode.shared.edits.dir>> - the URI which identifies the grou of JNs where the NameNodes will write/read edits
Typo: grou
> This is where one configures the addresses of the JournalNodes which the
> Standby NameNode uses to stay up-to-date with all the file system changes the
> Active NameNode makes
This seems to indicate that the SBN is the only node using this command. Perhaps better to say something like "the addresses of the JournalNodes which provide the shared edits storage, written to by the Active NameNode and read by the Standby NameNode." or somesuch?
> should be the a URI that uses the scheme "qjournal", where the scheme-specific
typo: "the a"
>
The value of this setting
> should be the a URI that uses the scheme "qjournal", where the scheme-specific
> part is a semicolon-separated list of <host:port> addresses where the
> JournalNodes can be contacted, and the path component is a unique identifier for
> this nameservice. Though not a requirement, it's a good idea to reuse the
> nameservice ID for the journal identifier.
Though this is correct, I find it hard to read. Perhaps it would be simpler to say:
The URI should be of the form: qjournal://<host1:port1>;<host2:port2>;<host3:port3>/<journalId>. The Journal ID is a unique identifier for this nameservice, which allows a single set of JournalNodes to provide storage for multiple federated namesystems. Though not a requirement, it's a good idea to reuse the nameservice ID for the journal identifier.
(also keep the example paragraph that you have following this one)
> Thus, during a failover, we first ensure that the Active NameNode is either
> in the Standby state, or the process has terminated, before transitioning the
> other NameNode to the Active state. In order to do this, you must configure at
> least one <<fencing method.>> These are configured as a
> carriage-return-separated list, which will be attempted in order until one
> indicates that fencing has succeeded. There are two methods which ship with
> Hadoop: <shell> and <sshfence>. For information on implementing your own custom
> fencing method, see the <org.apache.hadoop.ha.NodeFencer> class.
This paragraph no longer seems relevant.
> Redundancy for this data is provided by running multiple separate JournalNodes
add: or configuring this directory on a locally attached RAID array.
> can be done by running the command "<hdfs journalnode>" and waiting for the
> daemon to start on each of the relevant machines.
Should probably have them use hdfs-daemon.sh instead so it daemonizes
> of your NameNode metadata directories to the other, unformatted NameNode
> using the command "<hdfs namenode -bootstrapStandby>". Running this command
Specify that this command should be run on the machine that will be the standby.
Perhaps it would be better to format this section as a step-by-step list.
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-3926.patch, qjm-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Aaron T. Myers (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Aaron T. Myers updated HDFS-3926:
---------------------------------
Attachment: regular-ha-doc.diff
qjm-ha-doc.diff
HDFS-3926.patch
Here's a patch which addresses the issue. This change forks the existing HA documentation into an NFS guide and a QJM guide. In doing so, it renames the regular guide to make it clear that it is for NFS. This change also takes the liberty of documenting the 'bootstrapStandby' and 'initializeSharedEdits" commands.
To ease review, I'm also attaching a small diff of the changes to the NFS HA doc (since the file was renamed), and a diff of the QJM HA doc against the previous HA doc, to make it clear what changes were required to make the doc QJM-specific.
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-3926.patch, qjm-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3926) QJM: Add user documentation for QJM
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3926?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13455396#comment-13455396 ]
Todd Lipcon commented on HDFS-3926:
-----------------------------------
> * <<dfs.namenode.shared.edits.dir>> - the URI which identifies the grou of JNs where the NameNodes will write/read edits
typo: 'grou' still
----
> the shared edits storage, written to by the Active nameNode and read by the
typo: 'nameNode"
----
> to use no actual fencing methods, you still must configure something for this
> setting, for example "<<</bin/true>>>".
Probably should say <<<shell(/bin/true)>>>
----
+1 with these addressed
> QJM: Add user documentation for QJM
> -----------------------------------
>
> Key: HDFS-3926
> URL: https://issues.apache.org/jira/browse/HDFS-3926
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: documentation
> Affects Versions: QuorumJournalManager (HDFS-3077)
> Reporter: Aaron T. Myers
> Assignee: Aaron T. Myers
> Attachments: HDFS-3926.patch, HDFS-3926.patch, qjm-ha-doc.diff, qjm-ha-doc.diff, regular-ha-doc.diff, regular-ha-doc.diff
>
>
> We should add user-facing documentation for how to configure/use the QJM.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira