You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "S.L." <sl...@gmail.com> on 2008/11/14 02:18:40 UTC
HDFS NameNode and HA: best strategy?
Hi list
I am kind of new to Hadoop but have some good background. I am seriously
considering adopting Hadoop and especially HDFS first to be able to store
various files (in the low hundreds thousands at first) on a few nodes in a
manner where I don't need a RAID system or a SAN. HDFS seems a perfect fit
for the job...
BUT
from what I learn in the past couple days it seems that the single point of
failure in HDFS is the NameNode. So I was wondering if anyone in the list
that did deploy HDFS in a production environment on what is their strategy
for High Availability of the system... Having the NameNode unavailable is
basically bringing the whole HDFS system offline. So what are the scripts or
other techniques recommended to add H.A to HDFS !
Thank !
-- S.
Re: HDFS NameNode and HA: best strategy?
Posted by Alex Loddengaard <al...@cloudera.com>.
The image and edits files are copied to the secondary namenode periodically,
so if you provision a new namenode from the secondary namenode, then your
new namenode may be lacking state that the original namenode had. You
should grab from the namenode NFS mount, not from the secondary namenode
image and edits files.
As for a script to do this, I'm not aware of one. However, it should be as
easy as a SCP or a RSYNC, a call to start-all.sh, etc.
Alex
On Fri, Nov 14, 2008 at 10:20 AM, Bill Au <bi...@gmail.com> wrote:
> There is a "secondary" NameNode which performs periodic checkpoints:
>
> http://wiki.apache.org/hadoop/FAQ?highlight=(secondary)#7
>
> Are there any instructions out there on how to copy the FS image and edits
> log from the secondary NameNode to a new machine when the original NameNode
> fails?
>
> Bill
>
> On Fri, Nov 14, 2008 at 12:50 PM, Alex Loddengaard <alex@cloudera.com
> >wrote:
>
> > HDFS does have a single point of failure, and there is no way around this
> > in
> > its current implementation. The namenode keeps track of a FS image and
> and
> > edits log. It's common for these to be stored both on the local disk and
> > on
> > a NFS mount. In the case when the namenode fails, a new machine can be
> > provisioned to be the namenode by loading the backed-up image and edits
> > files.
> > Can you say more about how you'll use HDFS? It's not a very latent file
> > system, so it shouldn't be used to serve images, videos, etc in a web
> > environment. It's most common use is to be the basis of batch Map/Reduce
> > jobs.
> >
> > Alex
> >
> > On Thu, Nov 13, 2008 at 5:18 PM, S. L. <sl...@gmail.com> wrote:
> >
> > > Hi list
> > > I am kind of new to Hadoop but have some good background. I am
> seriously
> > > considering adopting Hadoop and especially HDFS first to be able to
> store
> > > various files (in the low hundreds thousands at first) on a few nodes
> in
> > a
> > > manner where I don't need a RAID system or a SAN. HDFS seems a perfect
> > fit
> > > for the job...
> > >
> > > BUT
> > >
> > > from what I learn in the past couple days it seems that the single
> point
> > of
> > > failure in HDFS is the NameNode. So I was wondering if anyone in the
> list
> > > that did deploy HDFS in a production environment on what is their
> > strategy
> > > for High Availability of the system... Having the NameNode unavailable
> is
> > > basically bringing the whole HDFS system offline. So what are the
> scripts
> > > or
> > > other techniques recommended to add H.A to HDFS !
> > >
> > > Thank !
> > >
> > > -- S.
> > >
> >
>
Re: HDFS NameNode and HA: best strategy?
Posted by Bill Au <bi...@gmail.com>.
There is a "secondary" NameNode which performs periodic checkpoints:
http://wiki.apache.org/hadoop/FAQ?highlight=(secondary)#7
Are there any instructions out there on how to copy the FS image and edits
log from the secondary NameNode to a new machine when the original NameNode
fails?
Bill
On Fri, Nov 14, 2008 at 12:50 PM, Alex Loddengaard <al...@cloudera.com>wrote:
> HDFS does have a single point of failure, and there is no way around this
> in
> its current implementation. The namenode keeps track of a FS image and and
> edits log. It's common for these to be stored both on the local disk and
> on
> a NFS mount. In the case when the namenode fails, a new machine can be
> provisioned to be the namenode by loading the backed-up image and edits
> files.
> Can you say more about how you'll use HDFS? It's not a very latent file
> system, so it shouldn't be used to serve images, videos, etc in a web
> environment. It's most common use is to be the basis of batch Map/Reduce
> jobs.
>
> Alex
>
> On Thu, Nov 13, 2008 at 5:18 PM, S. L. <sl...@gmail.com> wrote:
>
> > Hi list
> > I am kind of new to Hadoop but have some good background. I am seriously
> > considering adopting Hadoop and especially HDFS first to be able to store
> > various files (in the low hundreds thousands at first) on a few nodes in
> a
> > manner where I don't need a RAID system or a SAN. HDFS seems a perfect
> fit
> > for the job...
> >
> > BUT
> >
> > from what I learn in the past couple days it seems that the single point
> of
> > failure in HDFS is the NameNode. So I was wondering if anyone in the list
> > that did deploy HDFS in a production environment on what is their
> strategy
> > for High Availability of the system... Having the NameNode unavailable is
> > basically bringing the whole HDFS system offline. So what are the scripts
> > or
> > other techniques recommended to add H.A to HDFS !
> >
> > Thank !
> >
> > -- S.
> >
>
Re: HDFS NameNode and HA: best strategy?
Posted by Alex Loddengaard <al...@cloudera.com>.
HDFS does have a single point of failure, and there is no way around this in
its current implementation. The namenode keeps track of a FS image and and
edits log. It's common for these to be stored both on the local disk and on
a NFS mount. In the case when the namenode fails, a new machine can be
provisioned to be the namenode by loading the backed-up image and edits
files.
Can you say more about how you'll use HDFS? It's not a very latent file
system, so it shouldn't be used to serve images, videos, etc in a web
environment. It's most common use is to be the basis of batch Map/Reduce
jobs.
Alex
On Thu, Nov 13, 2008 at 5:18 PM, S. L. <sl...@gmail.com> wrote:
> Hi list
> I am kind of new to Hadoop but have some good background. I am seriously
> considering adopting Hadoop and especially HDFS first to be able to store
> various files (in the low hundreds thousands at first) on a few nodes in a
> manner where I don't need a RAID system or a SAN. HDFS seems a perfect fit
> for the job...
>
> BUT
>
> from what I learn in the past couple days it seems that the single point of
> failure in HDFS is the NameNode. So I was wondering if anyone in the list
> that did deploy HDFS in a production environment on what is their strategy
> for High Availability of the system... Having the NameNode unavailable is
> basically bringing the whole HDFS system offline. So what are the scripts
> or
> other techniques recommended to add H.A to HDFS !
>
> Thank !
>
> -- S.
>
Re: NameNode does not come out of Safemode automatically in Hadoop-0.17.2
Posted by Pratyush Banerjee <pr...@aol.com>.
Thanks Lohit,
We were able to clean up the stuff with minimal data loss.
Pratyush
lohit wrote:
> Namenode does not come out of safemode until it gets confirmation from datanodes about the blocks it has. Namenode has a view of filesystem and its blocks, it expects those blocks to be reported by datanode until which it decides that the filesystem is not ready to use yet. You can exit the namenode from safemode and do 'dfs fsck / -files -blocks -locations' which will tell you the missing blocks and their locations where it expects from. Check if those nodes are up and running an they have those blocks.
> Thanks,
> Lohit
>
>
>
> ----- Original Message ----
> From: Pratyush Banerjee <pr...@aol.com>
> To: core-user@hadoop.apache.org
> Sent: Thursday, November 13, 2008 8:39:20 PM
> Subject: NameNode does not come out of Safemode automatically in Hadoop-0.17.2
>
> Hi All,
>
> We have been using hadoop-0.17.2 for some time now and we just had a case of namenode crash due to disk being full.
> In order to get the namenode up again with minimal loss of data, we had to manually edit the edits file in a Hex editor and restart the namenode.
>
> However after restarting, the namenode went to the safe mode (as expected), but it has been hours since it is like that, and it has not yet come out of the safemode.
> We can obviously force it to come out but should it not come out automatically ?
> Even after 12 hours of remaining in safemode the ratio of reported block size is still stuck at 0.9768.
>
> Running fsck on / in the hdfs does report about some corrupt files.
>
> What is the issue which is blocking namenode form coming out of safemode ? If we have to do it manually (hadoop dfsadmin -safemode leave) then what procedure do we follow in the process to ensure data safety ?
>
> thanks and regards,
>
> Pratyush
>
>
Re: NameNode does not come out of Safemode automatically in Hadoop-0.17.2
Posted by lohit <lo...@yahoo.com>.
Namenode does not come out of safemode until it gets confirmation from datanodes about the blocks it has. Namenode has a view of filesystem and its blocks, it expects those blocks to be reported by datanode until which it decides that the filesystem is not ready to use yet. You can exit the namenode from safemode and do 'dfs fsck / -files -blocks -locations' which will tell you the missing blocks and their locations where it expects from. Check if those nodes are up and running an they have those blocks.
Thanks,
Lohit
----- Original Message ----
From: Pratyush Banerjee <pr...@aol.com>
To: core-user@hadoop.apache.org
Sent: Thursday, November 13, 2008 8:39:20 PM
Subject: NameNode does not come out of Safemode automatically in Hadoop-0.17.2
Hi All,
We have been using hadoop-0.17.2 for some time now and we just had a case of namenode crash due to disk being full.
In order to get the namenode up again with minimal loss of data, we had to manually edit the edits file in a Hex editor and restart the namenode.
However after restarting, the namenode went to the safe mode (as expected), but it has been hours since it is like that, and it has not yet come out of the safemode.
We can obviously force it to come out but should it not come out automatically ?
Even after 12 hours of remaining in safemode the ratio of reported block size is still stuck at 0.9768.
Running fsck on / in the hdfs does report about some corrupt files.
What is the issue which is blocking namenode form coming out of safemode ? If we have to do it manually (hadoop dfsadmin -safemode leave) then what procedure do we follow in the process to ensure data safety ?
thanks and regards,
Pratyush
NameNode does not come out of Safemode automatically in Hadoop-0.17.2
Posted by Pratyush Banerjee <pr...@aol.com>.
Hi All,
We have been using hadoop-0.17.2 for some time now and we just had a
case of namenode crash due to disk being full.
In order to get the namenode up again with minimal loss of data, we had
to manually edit the edits file in a Hex editor and restart the namenode.
However after restarting, the namenode went to the safe mode (as
expected), but it has been hours since it is like that, and it has not
yet come out of the safemode.
We can obviously force it to come out but should it not come out
automatically ?
Even after 12 hours of remaining in safemode the ratio of reported block
size is still stuck at 0.9768.
Running fsck on / in the hdfs does report about some corrupt files.
What is the issue which is blocking namenode form coming out of
safemode ? If we have to do it manually (hadoop dfsadmin -safemode
leave) then what procedure do we follow in the process to ensure data
safety ?
thanks and regards,
Pratyush