You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "Brandon Li (JIRA)" <ji...@apache.org> on 2012/05/15 20:13:09 UTC
[jira] [Created] (HDFS-3425) Enable standby namenode to tail edits
from journal service
Brandon Li created HDFS-3425:
--------------------------------
Summary: Enable standby namenode to tail edits from journal service
Key: HDFS-3425
URL: https://issues.apache.org/jira/browse/HDFS-3425
Project: Hadoop HDFS
Issue Type: Sub-task
Components: ha, name-node
Reporter: Brandon Li
Assignee: Brandon Li
In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397014#comment-13397014 ]
Brandon Li commented on HDFS-3425:
----------------------------------
Todd, Thanks for explaining your approach in details. Using the same interface reading edit logs minimizes the code change on the namenode side. I planed to stream the edit logs in HDFS-3033. It's good that you already have it implemented in your branch.
I suspect you still need the "if(useJournalService())" alike calls when HA is considered later on. My understanding of the namenode start procedure with JournalService is like the following roughly:
1. namenode inits (in standby mode)
2. load fsimage
3. tail finalized segments
4. transit to active mode
5. collect last txnId from quorum number of JD (to find the txnId to roll edit)
6. roll edit, at east quorum number of JDs in sync
7. load the last (1 or 0) finalized edit segment
The namenode has to be aware of the HA approaches, shared edits or using journal service.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396447#comment-13396447 ]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
Hey Brandon,
I just got my implementation "working" (ie I can run a NN against remote loggers and then successfully restart reading the edits remotely). It's nowhere near finished/tested/pretty, but I pushed to github so I could share the approach:
This commit https://github.com/toddlipcon/hadoop-common/commit/c7aeeb83a653935463d0899968828abad3597ee8 refactors EditLogFileInputStream to deal with an interface instead of directly a {{File}} reference. The interface just has to be able to provide an input stream as well as its length and a name.
This commit https://github.com/toddlipcon/hadoop-common/commit/2969ab41478c808ff26ef3f00dad2cd22a352e93 adds an implementation of the above interface which reads from a remote URL, and changes the getEditLogManifest call to also expose the node's HTTP port (so it can use ephemeral ports)
Then this commit https://github.com/toddlipcon/hadoop-common/commit/477d937c258d4ffe67a5042258da7410df74c42e implements {{selectInputStreams()}} to actually pull the edits off the remote nodes while loading the namespace.
I'm not attacking HA yet in these patches, but the interfaces should work out the same there, given the BK folks managed to get it working.
As I said above, the code's rough, so I don't expect anyone to code review the github branch, but if you have questions on the approach, let me know. I'm trying to reuse as much of your JournalService code as possible, but mostly doing so by copy-pasting into these new files for now, to avoid conflicts, etc. I anticipate running back over it to unify the code better at a later date.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396962#comment-13396962 ]
Suresh Srinivas commented on HDFS-3425:
---------------------------------------
Quick comment (have not looked at either of the implementation). The code should be reusable and should include a component that can read edits from a JournalNode. That way the components can be used in all the following conditions:
- JournalNode, Namenode or a stand alone process reading journal from a given JN
- JN, NN or a stand alone process reading journal from a quorum of JNs
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397099#comment-13397099 ]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
bq. Todd, I think you are right. When can you commit your changes?
I'm hoping to have a fully-functional prototype by the end of this week - glad to finally have some coding time without conferences to worry about ;-)
After that I will do the work to split the changes into more digestible chunks rather than the big git branch. Since some of the pieces are still in flux I didn't want to try to start filing individual JIRAs for each item, only to later find out the interface wasn't quite right.
bq. As Suresh pointed out, the code should be usable by NN and JN to stream logs from each other (or just between JNs).
Yes, we need to also do the work so that the 2NN can work in a cluster where the NN has no local edits storage. This implies that the GetImageServlet either needs to read from the JournalManager API, or the 2NN needs to use the JM API to transfer edits. I'm not quite sure yet which is the right approach. Do you have any thoughts on that?
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396431#comment-13396431 ]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
bq. I am not sure I got the idea about using selectInputStreams() instead of if(usesJournalService()). Is your branch currently visible, so we can see an example?
That's what I'm working on right now. But basically, the idea is the following:
- the NN runs with just a QuorumJournalManager, which maintains the list of remote JournalNodes to talk to. I believe your design doc calls this a JournalList instead, but same thing.
- when the NN (or the Standby NN) wants to read edits, it uses editLog.selectInputStreams() exactly as it does today. But the QuorumJournalManager implementation of it services the call as follows:
-- RPC to each of the remote journals
-- get back a list of finalized edit log segments
-- for each, creates an object which stores the HTTP URL at which that segment is accessible, along with the txid info
-- creates an EditLogFileInputStream for each such object, but when opened, instead of creating FileInputStream, it opens the URL connection and calls getInputStream() on the resulting URL
So, the NN code remains identical, and all the magic is encapsulated in the journal manager implementation. This is basically the same as how the BookKeeper implementation works today -- the journal manager just has to implement selectInputStreams and the rest of the code _should_ work the same without special cases.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397068#comment-13397068 ]
Brandon Li commented on HDFS-3425:
----------------------------------
Todd, I think you are right. When can you commit your changes?
As Suresh pointed out, the code should be usable by NN and JN to stream logs from each other (or just between JNs).
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3425) Enable standby namenode to tail edits
from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3425:
-----------------------------
Attachment: HDFS-3425.HDFS-3092.patch.2
re-base the patch with HDFS-3092 head.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396426#comment-13396426 ]
Brandon Li commented on HDFS-3425:
----------------------------------
Hi Todd, thanks for reviewing it. One implementation here is better than two. :-)
I am not sure I got the idea about using selectInputStreams() instead of if(usesJournalService()). Is your branch currently visible, so we can see an example?
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397124#comment-13397124 ]
Brandon Li commented on HDFS-3425:
----------------------------------
{quote}Yes, we need to also do the work so that the 2NN can work in a cluster where the NN has no local edits storage.{quote}
I assume you talk about the other standby namenode.
Without considering checkpointing, the 2NN is in step 3 (tailing finalized segments) in your list. So if we use JM API for NN in steps 3 to 7, it seems natural to me to use the same JM API for the 2NN. Did I miss anything here?
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13397029#comment-13397029 ]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
My plan is the following (similar to what you said:)
3. tail finalized segments
4. transit to active mode
5. calls recoverUnclosedStreams()
5a. Does the "recovery" process whereby it determines the correct txn to synchronize to
5b. finalizes the last segment to this txnid on all the available JNs
6. reads the last finalized segment
7. begins writing at the next txid
I think this interface can be kept entirely generic -- recoverUnfinalizedSegments() is already a JM API which is implemented correctly by FileJournalManager.
Looking at {{startActiveServices()}} we basically already take this approach of calling {{recoverUnclosedStreams()}} followed by tailing edits one more time before writing.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Updated] (HDFS-3425) Enable standby namenode to tail edits
from journal service
Posted by "Brandon Li (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Brandon Li updated HDFS-3425:
-----------------------------
Attachment: HDFS-3425.HDFS-3092.patch
This is the initial patch to show the approach and the scope of changes.
To tail edits, SNN downloads finalized segments from journal service to storage directories and then load them. Here I try to minimize the change on NN and refuse the same methods used by journal sync.
It is expected to be improved later on by directly streaming the edit logs from journal service to SNN. The security change in Http server will be addressed in a different JIRA HDFS-3426.
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] [Commented] (HDFS-3425) Enable standby namenode to tail
edits from journal service
Posted by "Todd Lipcon (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-3425?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13396403#comment-13396403 ]
Todd Lipcon commented on HDFS-3425:
-----------------------------------
Hey Brandon,
I'm just looking over this quickly, and it seems inelegant that we have a lot of "if (usesJournalService())" type calls. In the code I'm working on, this is all done using the existing interfaces like selectInputStreams() so we don't need to have two implementations of all this code.
I'm also working on the "direct-from-HTTP" loading. I'll try to push my branch to github later, but it's currently in a lot of flux so I don't want to try to separate it out into clean JIRAs before its' a bit farther along. Is that OK?
> Enable standby namenode to tail edits from journal service
> ----------------------------------------------------------
>
> Key: HDFS-3425
> URL: https://issues.apache.org/jira/browse/HDFS-3425
> Project: Hadoop HDFS
> Issue Type: Sub-task
> Components: ha, name-node
> Reporter: Brandon Li
> Assignee: Brandon Li
> Attachments: HDFS-3425.HDFS-3092.patch, HDFS-3425.HDFS-3092.patch.2
>
>
> In an HA cluster with the journal service configured, namenode should get edit logs from journal service instead of from local or shared storage directories.
> As one of the first steps towards this goal, this JIRA is to enable standby namenode to tail edits from journal services.
--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira