You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-issues@hadoop.apache.org by "dhruba borthakur (JIRA)" <ji...@apache.org> on 2010/02/13 11:36:28 UTC
[jira] Created: (HDFS-976) Hot Standby for NameNode
Hot Standby for NameNode
------------------------
Key: HDFS-976
URL: https://issues.apache.org/jira/browse/HDFS-976
Project: Hadoop HDFS
Issue Type: New Feature
Components: name-node
Reporter: dhruba borthakur
Assignee: dhruba borthakur
This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Assigned: (HDFS-976) Hot Standby for NameNode
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur reassigned HDFS-976:
-------------------------------------
Assignee: Dmytro Molkov (was: dhruba borthakur)
I had an offline discussion with Dmytro. He will upload a new version of this patch (with details on how to configure it).
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: AvatarNode.patch, AvatarNodeDescription.txt
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Commented: (HDFS-976) Hot Standby for NameNode
Posted by "aleojiang (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12855931#action_12855931 ]
aleojiang commented on HDFS-976:
--------------------------------
Would you please provide the steps and configurations on how to do test on your patch?
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: AvatarNode.patch, AvatarNodeDescription.txt
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
If you think it was sent incorrectly contact one of the administrators: https://issues.apache.org/jira/secure/Administrators.jspa
-
For more information on JIRA, see: http://www.atlassian.com/software/jira
[jira] Commented: (HDFS-976) Hot Standby for NameNode
Posted by "Mingjie Lai (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=12878318#action_12878318 ]
Mingjie Lai commented on HDFS-976:
----------------------------------
@Dmytro
Which branch did you make the patch against?
I checked out hadoop-common branch-0.20, and had been trying to apply your patch, but always got compilation errors -- http://pastebin.ca/1881910.
I'm pretty new to the hdfs code base. Did I miss anything here?
compile:
[echo] contrib: highavailability
[javac] Compiling 10 source files to /home/mlai/git/hadoop/hadoop-common/build/contrib/highavailability/classes
[javac] /home/mlai/git/hadoop/hadoop-common/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs/AvatarClient.java:258: init() has protected access in org.apache.hadoop.fs.FsShell
[javac] fh.init();
[javac] ^
[javac] /home/mlai/git/hadoop/hadoop-common/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs/AvatarClient.java:260: delete(org.apache.hadoop.fs.Path,org.apache.hadoop.fs.FileSystem,boolean,boolean) has private access in org.apache.hadoop.fs.FsShell
[javac] fh.delete(p, p.getFileSystem(conf), recursive, false);
[javac] ^
[javac] /home/mlai/git/hadoop/hadoop-common/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs/AvatarClient.java:1849: cannot find symbol
[javac] symbol : class BlockMissingException
[javac] location: class org.apache.hadoop.hdfs.AvatarClient.DFSInputStream
[javac] throw new BlockMissingException(src, "Could not obtain block: " + blockInfo, block.getStartOffset());
[javac] ^
[javac] /home/mlai/git/hadoop/hadoop-common/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs/server/datanode/AvatarDataNode.java:659: cannot find symbol
[javac] symbol : method getNameNodeAddress(org.apache.hadoop.conf.Configuration)
[javac] location: class org.apache.hadoop.hdfs.server.datanode.DataNode
[javac] return DataNode.getNameNodeAddress(newconf);
[javac] ^
[javac] /home/mlai/git/hadoop/hadoop-common/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs/server/namenode/AvatarNode.java:291: cannot find symbol
[javac] symbol : method writeLock()
[javac] location: class org.apache.hadoop.hdfs.server.namenode.FSNamesystem
[javac] namesystem.writeLock();
...
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: AvatarNode.patch, AvatarNodeDescription.txt, AvatarPatch.2.patch
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-976) Hot Standby for NameNode
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-976:
-------------------------------
Attachment: AvatarNode.20.patch
Sorry, the last patch was made agains our internal tree.
This one is modified to be applied agains hadoop-0.20 trunk.
I will be working on creating a patch against trunk, but it will take some time. In the meanwhile this one seems to compile OK against hadoop-0.20
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: AvatarNode.20.patch, AvatarNode.patch, AvatarNodeDescription.txt, AvatarPatch.2.patch
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-976) Hot Standby for NameNode
Posted by "Dmytro Molkov (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Dmytro Molkov updated HDFS-976:
-------------------------------
Attachment: AvatarPatch.2.patch
I am attaching a newer version of the patch.
We fixed a few issues that were leading to data corruption and are actively testing this version to see if there are any more problems.
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: Dmytro Molkov
> Attachments: AvatarNode.patch, AvatarNodeDescription.txt, AvatarPatch.2.patch
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.
[jira] Updated: (HDFS-976) Hot Standby for NameNode
Posted by "dhruba borthakur (JIRA)" <ji...@apache.org>.
[ https://issues.apache.org/jira/browse/HDFS-976?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
dhruba borthakur updated HDFS-976:
----------------------------------
Attachment: AvatarNode.patch
AvatarNodeDescription.txt
A short writeup and code to implement a Hot failover for NameNode in hadoop 0.20. This code is by no means complete and extensive testing is required before it can be deployed. It is developed purely as a contrib module so that the HA functionality is completely separated from the existing NameNode.
> Hot Standby for NameNode
> ------------------------
>
> Key: HDFS-976
> URL: https://issues.apache.org/jira/browse/HDFS-976
> Project: Hadoop HDFS
> Issue Type: New Feature
> Components: name-node
> Reporter: dhruba borthakur
> Assignee: dhruba borthakur
> Attachments: AvatarNode.patch, AvatarNodeDescription.txt
>
>
> This is a place holder to share our code and experiences about implementing a Hot Standby for the HDFS NameNode for hadoop 0.20.
--
This message is automatically generated by JIRA.
-
You can reply to this email to add a comment to the issue online.