You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-dev@hadoop.apache.org by "Konstantin Shvachko (JIRA)" <ji...@apache.org> on 2008/10/29 20:20:44 UTC
[jira] Created: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Streaming Edits to a Standby Name-Node.
---------------------------------------
Key: HADOOP-4539
URL: https://issues.apache.org/jira/browse/HADOOP-4539
Project: Hadoop Core
Issue Type: New Feature
Components: dfs
Reporter: Konstantin Shvachko
Fix For: 0.20.0
Attachments: StreamEditsToSNN.htm
Currently Secondary name-node acts as mere checkpointer.
Secondary name-node should be transformed into a standby name-node (SNN).
The long term goal is to make it a warm standby.
The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Resolution: Fixed
Status: Resolved (was: Patch Available)
I just committed this.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643648#action_12643648 ]
dhruba borthakur commented on HADOOP-4539:
------------------------------------------
Cool docs! One quick question: Does this design allow for multiple simultaneous Stabdby-NameNode?
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Fix Version/s: 0.21.0
Status: Patch Available (was: Open)
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682529#action_12682529 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
DatanodeCommand and DatanodeRegistration got modified, but their serialization did not change so it will work with the old versions.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681937#action_12681937 ]
Hadoop QA commented on HADOOP-4539:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402164/BackupNode.patch
against trunk revision 753346.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
-1 core tests. The patch failed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/86/console
This message is automatically generated.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: StreamEditsToBN.pdf
This is the updated document.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680719#action_12680719 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
I am going to comment on Suresh's items that are not reflected in the new patch.
Design related:
1. Active node does not turn into Backup node in current design. We plan to have a possibility to turn a Standby node into Active, yes but this is done by just changing the role of the node. I think it makes sense to keep BackupNode as a subclass of NameNode because some commands, those that don't modify the meta-data, will work on the backup node exactly as they do on the active node. Say you can do {{lsr}} from the backup, but if you try to create a file it will throw {{SafeModeException}}.
2. EditLogBackupOutputStream failing to write to the backup node is handled by calling {{processIOError()}}, which removes the failed BackupOutputStream from the list of EditLogOutputStream(s) and increments the checkpointTime for the remaining streams to make the failed image outdated.
Coding related:
1.1. name-node and data-node vs NameNode and DataNode is actually consistent if you look at other comments. Lets say NameNode and DataNode refer to classes and name-node and data-node to the abstractions behind them.
1.2. Calling super() in class constructors is unnecessary, but helps me to debug and analyze the code. You can set a breakpoint, step into, and easily find the code of the super constructor.
2.3. {{UnregisteredNodeException}} Constructor takes DatanodeID and DatanodeInfo as parameters because this is a constructor specific for data-nodes. We may later need name-node, balancer, etc specific constructors, which will have their special parameters.
5.1. We do not keep comments for previous versions in the code, they can be retrieved from svn history. I copied the respective comment from ClientProtocol.
6.1. I think that {{ServerCommand}} actions (static ints) should belong to the commands that use them. I plan to file an issue to move {{DatanodeCommand}} actions into the respective commands.
7.1. My reasoning is that we have NameNode, DataNode, and now we will have Backup Node and Checkpoint node. I first thought about naming them "Backup NameNode" and "Checkpoint NameNode" connotative to SecondaryNameNode.
But "Backup node" sounds better to me and besides this is what they are - nodes.
8.3. {{dfs.backup.address}} and {{dfs.backup.http.address}} define the addresses which the backup node starts on its rpc and http servers. You do not need to change the configuration when the node transitions from backup to active.
Example. If you start DataNode on port 0, which means on any free port, you do not change the config file after startup, right? Same with backup/active nodes.
8.9. Backup node does not have backup streams now, so {{errorReport()}} is just harmless. If we decide to implement chaining edits streams then yes we will need to override {{errorReport()}} and probably other methods too.
8.10. {{BackupNode.journalSize()}} will report the correct size of its edits file. I don't think it should throw an exception.
8.11. {{BackupNode.stop()}} should call {{RPC.stopProxy()}} because it needs to stop the name-node proxy it is connected to as a client.
And the {{NameNode.stop()}} will not take care of that because NameNode does not act as its own client.
9.1. {{Checkpointer.doCheckPoint()}} first calls {{NameNode.startCheckpoint()}} which in turn calls {{startJournalSpool()}} from the NameNode itself and does not return until the BackupNode actually rolls its edits.
9.2. {{convergeJournalSpool()}} contains common logic for Backup and Checkpoint nodes. I agree the name does not make much sense for the checkpoint node since it does not converge no journals as it doesn't have any.
But for the backup node its reflects the semantics of the operation.
9.3. {{Checkpointer}} should not be an inner class of the {{BackupNode}}, imo. We have been through this with FSNamesystem class, had to factor out most of the stuff out of it.
9.4. {{downloadCheckpoint()}} and {{uploadCheckpoint()}} should evelve in the future. Right now they use {{CheckpointSignature}} for validation, but we may need to include registration as well.
11.1. {{CheckpointSignature}} currently includes the name-nodes's edits file modification time only for identification purposes. I planned to create an issue to include there the number of files in the namepsace at the time when the checkpoint starts.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: StreamEditsToSNN.htm
This is the design document for review.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Suresh Srinivas updated HADOOP-4539:
------------------------------------
Hadoop Flags: [Reviewed]
+1 for the patch.
Minor nits that need not be fixed:
# {{HdfsConstants.java}} two spaces in {{"Standby Node"}}
# Findbugs warning can be removed by initializing {{myMetrics}} in {{NameNode.createNameNode()}} method
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12676437#action_12676437 ]
Suresh Srinivas commented on HADOOP-4539:
-----------------------------------------
I will send comments on the design doc later.
Comments (lot of them are nits...):
Design related:
1. Design uses BackupNode as a subclass of NameNode. In future where Backup can transition to Active or Active can transition to Backup, this relationship seems stringent. Capturing active, backup as the state of Namenode might be a better idea.
2. If EditLogBackupOutputStream fails to write to the backup, how does active and backup recover from this condition. Looks like the current mechanism to recover a failed stream to disk is not workable for stream to backup.
Coding related:
1. General comments:
1.1. For consistently use NameNode and DataNode intead of name-node and data-node
1.2. Many class contructors call default constructor of the super class {{super()}}. This is unnecessary.
1.3. Many if statements are not enclosed in curly brackets. Do we follow this java coding convention?
2. UnregisterNodeException.java
2.1. is shown as changed file. It should be new add and UnregisteredDatanodeException.java should be deleted.
2.2. Class comment still only talks about Datanode. It should be made generic to all types of node
2.3. Constructor can be change to take {storageID} and {name} instead of {DatanodeID} and {DatanodeInfo}
3. NodeRegistration.java
3.1. Class comment seems to indicate this is NameNode specific. Comment should indicate that it is generic.
4. CheckpointCommand.java
4.1. Additional description for the class to indicate that this is sent in response to {{startCheckpoint}} NamenodeProtocol command would be good.
5. NamenodeProtocol.java
5.1. In class comments why are the changes made for previous protocol version removed (in this case version 2). Isn't retaining this history a good idea for better documentation?
5.2. {{getBlocks()}} comment suggests that it i used by {{Balancer}}. Why have this comment if this could be used by things other than {{Balancer}} in the future.
5.3. Description for methods {{versionRequest}}, {{errorReport}}, {{endCheckpoint}},
5.4. {{startCheckpoint()}} good to indicate direction of the message (from backup to active)
5.5. Move static variables for Journal pipeline action to the beginning of the class
5.6. {{journal()}} better documentation on what it does and the direction of this message.
6. ServerCommand.java
6.1. Should the action static ints be moved to NamenodeProtocol.java? This is similar to Journal actions defined in NamenodeProtocol.java
7. HdfsConstants.java
7.1. It is better to be explicit about printing {{NamenodeRole}} as "Backup NameNode" and "Checkpoint NameNode" instead of just "Backup Node" and "Checkpoint Node".
8. BackupNode.java
8.1. In class comments </li> and </ol> are missing.
8.2. No need to defined {{LOG}} as it is inherited from {{NameNode}}
8.3. Using dfs.backup.address and dfs.backup.http.address means backup is defined at the install time. To change backup to active namenode, configuration change needs to be made. This could be restrictive in the future when a backup node can automatically transtion to become active.
8.4. When starting nodes, we may need to check the address read from configuration with the local node address for http.address etc.
8.5. Comment "NameNodeCommon methods" should change to "Common NameNode methods"
8.6. {{import org.apa/che.commons.logging.*;}} is not used
8.7. Replace {{import java.io.*; import java.net.*}} to import specific ones {{IOException, InetSocketAddress, SocketTimeoutException}}
8.8. {{handleShake()}} use {{NamenodeProtocol.NOTIFY}} instead of {{DatanodeProtocol.NOTIFY}} in {{errorReport}}
8.9. Should BackupNode override {{Namenode.errorReport()}}, in order to not do the processing of releasing backup node?
8.10. Should BackupNode throw exception for NamenodeProtocol methods {{NameNode.journalSize()}}?
8.11. {{stop()}} - is there a need for {{RPC.stopProxy(namenode)}}. Does {{super.stop()}} take care of it?
9. Checkpointer.java
9.1. {{doCheckPoint()}} calls {{backupNode.setCheckpointState(CheckpointStates.ROLLED_EDITS)}}. Should we do this in {{startJournalSpool()}}. That way for some reason {{startCheckpoint()}} to active fails, the checkpointer state remains consistent. {{doCheckPoint()}} should wait for state to reach CheckpointStates.ROLLED_EDITS to make sure the active has rolled edits before proceeding further with checkpoint. This helps in any boundary conditions where checkpoint could miss edits that might still come from active to previous edits log.
9.2. {{doCheckpoint()}} should call {{convergeJournalSpool()}} for BackupNode of type BACKUP only
9.3. Given that Checkpointer is looking into internals of BackupNode so much, should it be made private inner class of BackupNode?
9.4. Should {{downloadCheckpoint()}} and {{uploadCheckpoint()}} have registrationID or some other token that is ephemeral perhaps for validating the access. Otherwise these URLs could be misused to mess with the Namenode state. This could be done in another jira.
9.5. {{run()}} Use either {{period}} or {{checkpointPeriod}} in the while loop. Also we can keep {{1000 * period}} calculated outside the while loop.
10. NameNode.java
10.1. For better readability NameNode could support instead of {{node.getRole() == NamenodeRole.BACKUP}}, method {{isRole(NamenodeRole.BACKUP)}} or {{isActive()}}, {{isBackup()}}, {{isCheckpointer()}}.
10.2. Not sure what the comment {{// trash}} in {{NameNode}} constructor is far...
11. CheckpointSignature.java
11.1. Need to create a separate jira to enhance this to include additional information that could help in catching error conditions such as backup node FSImage number of files not matching the active node etc. Currently any condition where backup does not match the active goes undetected.
12 EditLogBackupInputStream.java
12.1. Should constructor be using parameter {{name}} to set the member {{address}}?
12.2. {{getType()}} should return {{JournalType.BACKUP}}
12.3. Class comments need to be updated
13. FSNamesystem.java
13.1. {{getStorageDirs()}} {{FSNamesystem.LOG}} can just be {{LOG()}}
14. FSImage.java
14.1. minor nit - {{startCheckpoint()}} comment has name-name.
14.2. {{startCheckpoint()}} should some of the {{if}} conditions be {{else if}} where {{msg}} is set
14.3. Adding parentheses to the if condition that combines || with && will make it more readable
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682498#action_12682498 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4539:
------------------------------------------------
- This is an incompatible change
- Need release notes
- Need to updated DatanodeProtocol version since DatanodeCommand is changed.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681924#action_12681924 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
S> ACT_SHUTDOWN is part of protocol itself, it should be in NamenodeProtocol.java
Makes sense I moved all action codes into {{NamenodeProtocol}}.
About your address concerns: you in fact propose a change to how configuration works.
You are saying that configuration should contain addresses {{rcp.address}} and {{http.address}} which would be used by the server starting on this node no matter what this server is. This is different from how things are in hadoop now. With your approach you will have to create a separate configuration for each server. Say for the name-node you will need to set local {{rcp.address = X}} and {{http.address = Y}}, while for data-nodes you will have to move this values to other variables {{namenode.rcp.address = X}} and {{namenode.http.address = Y}} because {{rcp.address}} and {{http.address}} should reflect the data-nodes' addresses.
We on the contrary were trying so far (as much as possible) to keep one configuration for all servers: set everything in one config file and ssh it to all servers and clients unmodified.
Well this task is not for this issue anyway.
N> Static variables sometimes are misused as global variables.
Exactly!
N> I suggest passing a FSNamesystem object in the constructor BackupStorage().
It's the other way around: Backup node constructs {{BackupStorage()}} first then creates {{FSNamesystem}} with the BackupStorage as a parameter.
My idea with the comment you cited above was that while fixing HADOOP-5119 we will make a member in FSImage referencing {{FSNamesystem}}, and a getter {{getFSNamesystem()}}, which returns the reference. Then {{BackupStorage()}} will require only one change - removing the method you cited.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675084#action_12675084 ]
dhruba borthakur commented on HADOOP-4539:
------------------------------------------
Hi Konstantin, this is very interesting. I will wait for your design doc. A few short questions in the meantime:
1. Will this support having multiple Backup nodes for a single namenode?
2. What are the pros-and-cons of this effort vs the Bookkeeper integration (HADOOP-5189)?
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12675151#action_12675151 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
(Dhruba-1) Current implementation supports only one Backup node. It will just not let the other register once one is already registered. This is the implementation restriction only. We can extend it by providing a Collection of registered backup nodes on the active node. Something to work on in the future. You can have multiple checkpointers though as long as there are no any backups.
(Dhruba-2) Bookkeeper integration is a separate issue but related. Separate because BK provides only the journalling functionality. You cannot keep an up-to-date namespace in BK, and even if you do it is not useful for the name-node. Separate because fast journalling is important and that is what BK provides, while backup nodes are still bounded by the local disk speed. Related since both should use the same abstractions for journalling - abstract edit streams under the common journalling framework.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Status: Patch Available (was: Open)
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643649#action_12643649 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
Yes you should be able to run multiple standby nodes by design. NN will stream the journal to all of them.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
This patch introduces two new types of name-nodes: a Checkpoint node and a Backup node.
- The role of the *Checkpoint node* to checkpoint name-node meta-data by merging image and edits files.
- The *Backup node* extends functionality of the Checkpointer by that it can receive online updates of the file system meta-data, apply them to its memory state and persist them on disks just like the name-node does. Thus at any time the Backup node contains an up-to-date image of the namespace both in memory and on local disk(s).
This also results in much more efficient checkpointing because backup node does not need to transfer files from the active name-node and does not need to replay (merge) edits.
- Term *Standby node* is reserved for further extension of the backup node functionality, when cluster will be able to switch over to the new name-node if the active dies.
This is mentioned in the "Warm standby provision" section of the design document.
Typical use cases:
# Run Checkpoint node only to create checkpoints. This should be used instead of the current SecondaryNameNode, which is depricated by the patch. I reused a lot of the SecondaryNameNode code so this effort was not wasted, it just evolved.
# Run Backup node to support online streaming of edits and efficient checkpointing.
This particularly targets eliminating NFS as a remote storage for edits.
# Run NameNode without persistent storage at all and delegate all "persisting" functionality to the Backup node. The trick here is to start name-node with {{-importCheckpoint}} option and then run the Backup node.
In the near term I plan to
- attach an updated design document with all modifications and clarifications to the initial design.
- provide more test cases in TestBackupNode unit test;
- and perform large scale testing.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
Synced with trunk and removed space in "Standby Node".
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Status: Patch Available (was: Open)
Forgot to mention that I fixed 3 old find bugs, so the overall count went down by 2.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Tsz Wo (Nicholas), SZE (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681833#action_12681833 ]
Tsz Wo (Nicholas), SZE commented on HADOOP-4539:
------------------------------------------------
> In general I don't also see a reason why NameNode.myMetrics should be a static member.
Static variables sometimes are misused as global variables. This is one example. Another one is FSNamesystem.fsNamesystemObject (see HADOOP-2413).
BTW, could you not using the deprecated API FSNamesystem.getFSNamesystem() in BackupStorage?
{noformat}
+ private FSNamesystem getFSNamesystem() {
+ // HADOOP-5119 should get rid of this.
+ return FSNamesystem.getFSNamesystem();
+ }
{noformat}
I suggest passing a FSNamesystem object in the constructor BackupStorage(). Then, pass the FSNamesystem object created in BackupNode.loadNamesystem(..), which is the only place creating a BackupStorage.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Status: Open (was: Patch Available)
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: (was: BackupNode.patch)
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Status: Open (was: Patch Available)
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Release Note:
Introduced Backup node, which maintains an up-to-date state of the namespace in sync with the name-node.
Introduced Checkpoint node, which creates checkpoints of the namespace state. This is a replacement for SecondaryNameNode.
Hadoop Flags: [Incompatible change, Reviewed] (was: [Reviewed])
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681624#action_12681624 ]
Hadoop QA commented on HADOOP-4539:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12402109/BackupNode.patch
against trunk revision 753052.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 1 new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-minerva.apache.org/59/console
This message is automatically generated.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: image001.gif
Here is the missing image.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12644134#action_12644134 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
Most answers are in the document. I'll try to clarify.
# _Multiple standby nodes can be connected to the same active name-node._
Journal spool is an internal SNN mechanism that helps to delay applying journal records to the SNN's memory namespace while SNN starts up or creates a checkpoint. So if you have multiple standbys each of them will support its own journal spool. Don't see a problem here.
The question is how the name-node handles rollEdits from the second SNN when the checkpointing for the first one is still in progress. There are 2 ways to handle this.
## Declare that NN can tolerate only one checkpoint at a time and refuse checkpoint requests from other standbys until the current is complete. This is the simplest approach, and I plan to implement it at least initially.
## NN can create multiple edits.new for each subsequent rollEdits request. That is, when SNN1 asks NN to rollEdits NN creates edits.new.1
When SNN2 asks NN to rollEdits and edits.new.1 still exists NN creates edits.new.2 and so on. This is more tricky but is doable too.
# _SNN treats each received journal record as a name-node command._
Journal records in hdfs are self-contained, that is they do not depend on the current state of the name-server. Say, if a modification time should be changed then the time to which it should be changed to is specified in the record, it is not taken from SNN.system.currentTime().
# When SNN receives a record it applies it to the namespace in its memory AND if any storage directories are specified for SNN it will write the record to the edits file(s), just as NN would do, AND THEN will reply to the name-node that operation is complete.
# Journal spool persistently stores records on a local disk.
# _NN should keep running even if the last storage directory fails. It should enter safe mode in this case._
When all journal streams fail on the name-node it (NN) has no more way to persist namespace modifications. Currently we treat that by shutting down the name-node. Shutting down NN when the last hard drive fails does not work well since we loose both the image file and the in-memory state. I think we should keep NN running, but prevent it from name-space mutations. The safe-mode prevents it from changes. Keeping it in running condition reserves a chance to save the in-memory image to some other device manually mounted before calling saveCheckpoint.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12643684#action_12643684 ]
Suresh Srinivas commented on HADOOP-4539:
-----------------------------------------
Comments organized by section:
General comments:
- More description on how the NN and SNN startup and communicate with each other would be good to have.
- Design will only work for one standby namenode. For example if a standby node starts spooling, followed by another standby node how does flow work?
Streaming:
- Proposal is for SNN to treat each received record as name-node command and to call respective name-node method to update namespace. This is a not a good idea. For example modification time etc. which are based on NN will be different on SNN. SNN should get indication of what changed and apply it to its copy of namespace.
- How is the reliability of edit changes ensured? That is, when NN sends records to SNN how are the following ensured?
- Did SNN application really receive the record?
- Did SNN apply that record to its namespace copy?
- Did SNN store the the record in secondary storage reliably?
Journal Spool:
- Clarify that JS is stored as a file on the harddisk
Admin command saveCheckpoint
Not very clear on the following sentence. Does NN keep running if all the storage directories fail? Why should it then enter safe mode?
- NN should keep running even if the last storage directory fails. It should enter safe mode in this case.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Fix For: 0.20.0
>
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681968#action_12681968 ]
Konstantin Shvachko commented on HADOOP-4539:
---------------------------------------------
Testing comments.
- I tested this on a relatively small cluster at a rather heavy load.
- The test plan is the same as for the SecondaryNameNode, but
-# there are two nodes to test BackupNode and ChekpointNode
-# the registration procedure, which is new for BackupNode is tested by the unit test.
Thanks everybody for comments and help.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Hudson (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12682051#action_12682051 ]
Hudson commented on HADOOP-4539:
--------------------------------
Integrated in Hadoop-trunk #779 (See [http://hudson.zones.apache.org/hudson/job/Hadoop-trunk/779/])
. Introduce backup node and checkpoint node. Contributed by Konstantin Shvachko.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
I fixed all find bugs except for one. It is complaining about writing to a static field {{NameNode.myMetrics}}. This was introduced before. I don't see a reason why findBugs treats it differently now.
In general I don't also see a reason why {{NameNode.myMetrics}} should be a static member. I'll file a bug to change this.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Robert Chansler (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Robert Chansler reassigned HADOOP-4539:
---------------------------------------
Assignee: Konstantin Shvachko
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: image001.gif, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Suresh Srinivas (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12681830#action_12681830 ]
Suresh Srinivas commented on HADOOP-4539:
-----------------------------------------
1. I think it might be a good idea to abstract the active/backup/checkpointer as states of Namenode. This helps in transition of states back and forth. In current design an active node cannot become backup. This can be done in a separate jira.
Coding related:
bq. 6.1. I think that ServerCommand actions (static ints) should belong to the commands that use them. I plan to file an issue to move DatanodeCommand actions into the respective commands.
I am not clear on this. Since ACT_SHUTDOWN is part of protocol itself, it should be in NamenodeProtocol.java?
bq. 8.3. dfs.backup.address and dfs.backup.http.address define the addresses which the backup node starts on its rpc and http servers. You do not need to change the configuration when the node transitions from backup to active.
Example. If you start DataNode on port 0, which means on any free port, you do not change the config file after startup, right? Same with backup/active nodes.
My example is more like: Say you configure address X as active and address Y as backup during install time. Now for some reason you want to start active node on Y. At this time the addresses that Y tries to bind to address X. Similarly starting backup on X will result in X trying to bind to address Y. So irrespective of local address, the services start on configured addresses. I am not sure this can be solved until we have node management as a part of HA that can assign roles to the nodes. Nodes always bind to their local address and start or transition to the role they are assigned.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Konstantin Shvachko (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Konstantin Shvachko updated HADOOP-4539:
----------------------------------------
Attachment: BackupNode.patch
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Attachments: BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HADOOP-4539) Streaming Edits to a Standby
Name-Node.
Posted by "Hadoop QA (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HADOOP-4539?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12680741#action_12680741 ]
Hadoop QA commented on HADOOP-4539:
-----------------------------------
-1 overall. Here are the results of testing the latest attachment
http://issues.apache.org/jira/secure/attachment/12401793/BackupNode.patch
against trunk revision 752292.
+1 @author. The patch does not contain any @author tags.
+1 tests included. The patch appears to include 20 new or modified tests.
+1 javadoc. The javadoc tool did not generate any warning messages.
+1 javac. The applied patch does not increase the total number of javac compiler warnings.
-1 findbugs. The patch appears to introduce 9 new Findbugs warnings.
+1 Eclipse classpath. The patch retains Eclipse classpath integrity.
+1 release audit. The applied patch does not increase the total number of release audit warnings.
+1 core tests. The patch passed core unit tests.
+1 contrib tests. The patch passed contrib unit tests.
Test results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/testReport/
Findbugs warnings: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/artifact/trunk/build/test/findbugs/newPatchFindbugsWarnings.html
Checkstyle results: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/artifact/trunk/build/test/checkstyle-errors.html
Console output: http://hudson.zones.apache.org/hudson/job/Hadoop-Patch-vesta.apache.org/67/console
This message is automatically generated.
> Streaming Edits to a Standby Name-Node.
> ---------------------------------------
>
> Key: HADOOP-4539
> URL: https://issues.apache.org/jira/browse/HADOOP-4539
> Project: Hadoop Core
> Issue Type: New Feature
> Components: dfs
> Reporter: Konstantin Shvachko
> Assignee: Konstantin Shvachko
> Fix For: 0.21.0
>
> Attachments: BackupNode.patch, BackupNode.patch, BackupNode.patch, image001.gif, StreamEditsToBN.pdf, StreamEditsToSNN.htm
>
>
> Currently Secondary name-node acts as mere checkpointer.
> Secondary name-node should be transformed into a standby name-node (SNN).
> The long term goal is to make it a warm standby.
> The purpose of this issue is to provide real time streaming of edits to SNN so that it contained the up-to-date namespace state.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.