You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Ozcan ILIKHAN <il...@cs.wisc.edu> on 2011/04/23 07:17:26 UTC
HDFS + ZooKeeper
Hi,
Does anyone have any idea about how we can use HDFS with ZooKeeper? More
elaborately if NameNode fails DataNodes should be able to retrieve address
of new NameNode from ZooKeeper.
Thanks,
---------------------------------
Ozcan ILIKHAN
PhD Student, Graduate Research Assistant
Department of Computer Sciences
University of Wisconsin-Madison
http://pages.cs.wisc.edu/~ilikhan
Re: HDFS + ZooKeeper
Posted by Mag Gam <ma...@gmail.com>.
Will the avatar node be part of 0.22?
On Sat, Apr 23, 2011 at 6:29 PM, Dhruba Borthakur <dh...@gmail.com> wrote:
> The AvatarNode does use zookeeper (but since this is not directly related to
> Apache HDFS code, if u have more questions, please send it to me directly).
> The latest AvatarNode code is in
> https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs
> thanks,
> dhruba
>
> On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen
> <ja...@gmail.com> wrote:
>>
>> This is an interesting question. AvatarNode doesn't use Zookeeper?
>> Is there a drawback to implementing ZK for the NameNode other than the
>> potential bugs from a newly engineered feature?
>>
>> Perhaps this issue is the main one for implementing the NameNode in
>> ZK? HDFS-1623
>>
>> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
>> <jd...@apache.org> wrote:
>> > This is a 1M$ question. You could start thinking about this problem by
>> > looking at what "AvatarNode" does:
>> > https://issues.apache.org/jira/browse/HDFS-976
>> >
>> > J-D
>> >
>> > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu>
>> > wrote:
>> >> Hi,
>> >> Does anyone have any idea about how we can use HDFS with ZooKeeper?
>> >> More
>> >> elaborately if NameNode fails DataNodes should be able to retrieve
>> >> address
>> >> of new NameNode from ZooKeeper.
>> >>
>> >> Thanks,
>> >> ---------------------------------
>> >> Ozcan ILIKHAN
>> >> PhD Student, Graduate Research Assistant
>> >> Department of Computer Sciences
>> >> University of Wisconsin-Madison
>> >> http://pages.cs.wisc.edu/~ilikhan
>> >>
>> >
>
>
>
> --
> Connect to me at http://www.facebook.com/dhruba
>
Re: HDFS + ZooKeeper
Posted by Dhruba Borthakur <dh...@gmail.com>.
The AvatarNode does use zookeeper (but since this is not directly related to
Apache HDFS code, if u have more questions, please send it to me directly).
The latest AvatarNode code is in
https://github.com/facebook/hadoop-20-append/tree/master/src/contrib/highavailability/src/java/org/apache/hadoop/hdfs
thanks,
dhruba
On Sat, Apr 23, 2011 at 11:17 AM, Jason Rutherglen <
jason.rutherglen@gmail.com> wrote:
> This is an interesting question. AvatarNode doesn't use Zookeeper?
> Is there a drawback to implementing ZK for the NameNode other than the
> potential bugs from a newly engineered feature?
>
> Perhaps this issue is the main one for implementing the NameNode in
> ZK? HDFS-1623
>
> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
> <jd...@apache.org> wrote:
> > This is a 1M$ question. You could start thinking about this problem by
> > looking at what "AvatarNode" does:
> > https://issues.apache.org/jira/browse/HDFS-976
> >
> > J-D
> >
> > On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu>
> wrote:
> >> Hi,
> >> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
> >> elaborately if NameNode fails DataNodes should be able to retrieve
> address
> >> of new NameNode from ZooKeeper.
> >>
> >> Thanks,
> >> ---------------------------------
> >> Ozcan ILIKHAN
> >> PhD Student, Graduate Research Assistant
> >> Department of Computer Sciences
> >> University of Wisconsin-Madison
> >> http://pages.cs.wisc.edu/~ilikhan
> >>
> >
>
--
Connect to me at http://www.facebook.com/dhruba
Re: HDFS + ZooKeeper
Posted by Mahadev Konar <ma...@apache.org>.
Andy/others,
The BookKeeper work hasnt been completed yet. There is ongoing work in
https://issues.apache.org/jira/browse/HDFS-1580
and others.
Hopefully we'll have NN's running on BK soon!
thanks
mahadev
On Sun, Apr 24, 2011 at 10:20 AM, Andrew Purtell <ap...@apache.org> wrote:
>> From: Jason Rutherglen <ja...@gmail.com>
>> Right and AN is not using ZK for the actual NameNode
>> methods/functions, only for the failover election of the 'backup'
>> NameNode?
>
> What do you mean by "actual NameNode methods/functions"? I believe there was an effort during the development of AvatarNode to use BookKeeper (http://zookeeper.apache.org/doc/r3.3.3/bookkeeperOverview.html) to distribute the edit log. Is this what you mean?
>
> That effort ran into various problems but I don't know the detail.
>
> - Andy
>
>
--
thanks
mahadev
@mahadevkonar
Re: HDFS + ZooKeeper
Posted by Andrew Purtell <ap...@apache.org>.
> From: Jason Rutherglen <ja...@gmail.com>
> Right and AN is not using ZK for the actual NameNode
> methods/functions, only for the failover election of the 'backup'
> NameNode?
What do you mean by "actual NameNode methods/functions"? I believe there was an effort during the development of AvatarNode to use BookKeeper (http://zookeeper.apache.org/doc/r3.3.3/bookkeeperOverview.html) to distribute the edit log. Is this what you mean?
That effort ran into various problems but I don't know the detail.
- Andy
Re: HDFS + ZooKeeper
Posted by Jason Rutherglen <ja...@gmail.com>.
Right and AN is not using ZK for the actual NameNode
methods/functions, only for the failover election of the 'backup'
NameNode?
On Sun, Apr 24, 2011 at 4:06 AM, Ozcan ILIKHAN <il...@cs.wisc.edu> wrote:
> I have read Dhruba's blog entry about AvatarNode long ago. If I am not
> wrong initially it was not using ZooKeeper. That is way I raised that
> question. Also, AvatorNode is build on HDFS 0.20 but my intension is to
> utilize BackupNode that comes with HDFS 0.21
>
> Thanks,
> Ozcan.
>
> -----Original Message----- From: Jason Rutherglen
> Sent: Saturday, April 23, 2011 1:17 PM
> To: hdfs-user@hadoop.apache.org
> Subject: Re: HDFS + ZooKeeper
>
> This is an interesting question. AvatarNode doesn't use Zookeeper?
> Is there a drawback to implementing ZK for the NameNode other than the
> potential bugs from a newly engineered feature?
>
> Perhaps this issue is the main one for implementing the NameNode in
> ZK? HDFS-1623
>
> On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
> <jd...@apache.org> wrote:
>>
>> This is a 1M$ question. You could start thinking about this problem by
>> looking at what "AvatarNode" does:
>> https://issues.apache.org/jira/browse/HDFS-976
>>
>> J-D
>>
>> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu>
>> wrote:
>>>
>>> Hi,
>>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>>> elaborately if NameNode fails DataNodes should be able to retrieve
>>> address
>>> of new NameNode from ZooKeeper.
>>>
>>> Thanks,
>>> ---------------------------------
>>> Ozcan ILIKHAN
>>> PhD Student, Graduate Research Assistant
>>> Department of Computer Sciences
>>> University of Wisconsin-Madison
>>> http://pages.cs.wisc.edu/~ilikhan
>>>
>>
>
>
Re: HDFS + ZooKeeper
Posted by Ozcan ILIKHAN <il...@cs.wisc.edu>.
I have read Dhruba's blog entry about AvatarNode long ago. If I am not
wrong initially it was not using ZooKeeper. That is way I raised that
question. Also, AvatorNode is build on HDFS 0.20 but my intension is to
utilize BackupNode that comes with HDFS 0.21
Thanks,
Ozcan.
-----Original Message-----
From: Jason Rutherglen
Sent: Saturday, April 23, 2011 1:17 PM
To: hdfs-user@hadoop.apache.org
Subject: Re: HDFS + ZooKeeper
This is an interesting question. AvatarNode doesn't use Zookeeper?
Is there a drawback to implementing ZK for the NameNode other than the
potential bugs from a newly engineered feature?
Perhaps this issue is the main one for implementing the NameNode in
ZK? HDFS-1623
On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
<jd...@apache.org> wrote:
> This is a 1M$ question. You could start thinking about this problem by
> looking at what "AvatarNode" does:
> https://issues.apache.org/jira/browse/HDFS-976
>
> J-D
>
> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu>
> wrote:
>> Hi,
>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>> elaborately if NameNode fails DataNodes should be able to retrieve
>> address
>> of new NameNode from ZooKeeper.
>>
>> Thanks,
>> ---------------------------------
>> Ozcan ILIKHAN
>> PhD Student, Graduate Research Assistant
>> Department of Computer Sciences
>> University of Wisconsin-Madison
>> http://pages.cs.wisc.edu/~ilikhan
>>
>
Re: HDFS + ZooKeeper
Posted by Jason Rutherglen <ja...@gmail.com>.
This is an interesting question. AvatarNode doesn't use Zookeeper?
Is there a drawback to implementing ZK for the NameNode other than the
potential bugs from a newly engineered feature?
Perhaps this issue is the main one for implementing the NameNode in
ZK? HDFS-1623
On Fri, Apr 22, 2011 at 10:41 PM, Jean-Daniel Cryans
<jd...@apache.org> wrote:
> This is a 1M$ question. You could start thinking about this problem by
> looking at what "AvatarNode" does:
> https://issues.apache.org/jira/browse/HDFS-976
>
> J-D
>
> On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu> wrote:
>> Hi,
>> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
>> elaborately if NameNode fails DataNodes should be able to retrieve address
>> of new NameNode from ZooKeeper.
>>
>> Thanks,
>> ---------------------------------
>> Ozcan ILIKHAN
>> PhD Student, Graduate Research Assistant
>> Department of Computer Sciences
>> University of Wisconsin-Madison
>> http://pages.cs.wisc.edu/~ilikhan
>>
>
Re: HDFS + ZooKeeper
Posted by Jean-Daniel Cryans <jd...@apache.org>.
This is a 1M$ question. You could start thinking about this problem by
looking at what "AvatarNode" does:
https://issues.apache.org/jira/browse/HDFS-976
J-D
On Fri, Apr 22, 2011 at 10:17 PM, Ozcan ILIKHAN <il...@cs.wisc.edu> wrote:
> Hi,
> Does anyone have any idea about how we can use HDFS with ZooKeeper? More
> elaborately if NameNode fails DataNodes should be able to retrieve address
> of new NameNode from ZooKeeper.
>
> Thanks,
> ---------------------------------
> Ozcan ILIKHAN
> PhD Student, Graduate Research Assistant
> Department of Computer Sciences
> University of Wisconsin-Madison
> http://pages.cs.wisc.edu/~ilikhan
>