2 General questions - NameNode SPOF, Byzantine fault tolerance

[Maxim Veksler <ma...@vekslers.org>]

Hi,

New to Hadoop. Learning the architecture, code, ecosystem and everything
else... love what I've seen so far. Some parts are brilliant.

I've been wondering. If the NameNode is a single point of failure why not
employ a highly available solution to replace the current
implementation? For ex: Is there any reason why not to mimic the fsImage
file structure in Cassandra? or in ZooKeeper?

Another thing, I've only recently learned about BFT. Does hadoop uses
this algorithm somewhere in it's logic? I would love to see the
implementation if it does. Are there any plans for this? Seems like a good
fit for a large scale cluster.


Thanks for the continuous commits flow guys! :)
Great work.

Re: 2 General questions - NameNode SPOF, Byzantine fault tolerance

[Jeff Hammerbacher <ha...@cloudera.com>]

Hey Maxim,

Check out https://issues.apache.org/jira/browse/HDFS-1064 for the umbrella
JIRA for NN HA. For specific solutions, you may be interested in Facebook's
AvatarNode (https://issues.apache.org/jira/browse/HDFS-976), or China
Mobile's approach (http://github.com/gnawux/hadoop-cmri).

As you can see, multiple solution paths are being explored right now, and
your input would be valuable.

There are no BFT algorithms in the Hadoop codebase.

Thanks,
Jeff

On Tue, Aug 3, 2010 at 1:34 AM, Maxim Veksler <ma...@vekslers.org> wrote:

> Hi,
>
> New to Hadoop. Learning the architecture, code, ecosystem and everything
> else... love what I've seen so far. Some parts are brilliant.
>
> I've been wondering. If the NameNode is a single point of failure why not
> employ a highly available solution to replace the current
> implementation? For ex: Is there any reason why not to mimic the fsImage
> file structure in Cassandra? or in ZooKeeper?
>
> Another thing, I've only recently learned about BFT. Does hadoop uses
> this algorithm somewhere in it's logic? I would love to see the
> implementation if it does. Are there any plans for this? Seems like a good
> fit for a large scale cluster.
>
>
> Thanks for the continuous commits flow guys! :)
> Great work.
>