You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Ivan Tretyakov <it...@griddynamics.com> on 2014/01/31 17:26:02 UTC
Backup NameNode metadata in HA configuration
Hi!
We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
HDFS HA configuration is applied.
I would like to ask if there are any difference or possible side effects of
backing up namenode data directory.
It is ok to just copy NN data directory from active NN to another host,
what about standby NN?
Or it may NN metadate can be in inconsistent state?
Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Great! That is what I need to know. I was expecting that fsimage can be in
inconsistent state while applying changes and it can be wrong to just copy
it in such moments.
On Tue, Feb 4, 2014 at 11:42 AM, Harsh J <ha...@cloudera.com> wrote:
> Nothing wrong in copying it - except it would come with a bunch of
> redundant information (extra retained edit logs already merged into
> one of the recent fsimages).
>
> On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Thank you Harsh!
> > But I would like to know if just copying NN data directory (dfs.name.dir,
> > dfs.namenode.name.dir) is appropriate or not?
> >
> >
> > On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> You'll ideally just need the latest
> >> fsimage file, which includes the whole check pointed namespace, so
> >> "hdfs dfsadmin -fetch" is appropriate.
> >>
> >> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> >> <it...@griddynamics.com> wrote:
> >> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> >> > dfsadmin
> >> > -fetch', using NN HA configuration?
> >> >
> >> >
> >> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> >> > <it...@griddynamics.com> wrote:
> >> >>
> >> >> Hi!
> >> >>
> >> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >> >>
> >> >> HDFS HA configuration is applied.
> >> >>
> >> >> I would like to ask if there are any difference or possible side
> >> >> effects
> >> >> of backing up namenode data directory.
> >> >>
> >> >> It is ok to just copy NN data directory from active NN to another
> host,
> >> >> what about standby NN?
> >> >> Or it may NN metadate can be in inconsistent state?
> >> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>"
> command?
> >> >>
> >> >> --
> >> >> Best Regards
> >> >> Ivan Tretyakov
> >> >>
> >> >> Deployment Engineer
> >> >> Grid Dynamics
> >> >> +7 812 640 38 76
> >> >> Skype: ivan.v.tretyakov
> >> >> www.griddynamics.com
> >> >> itretyakov@griddynamics.com
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> > Ivan Tretyakov
> >> >
> >> > Deployment Engineer
> >> > Grid Dynamics
> >> > +7 812 640 38 76
> >> > Skype: ivan.v.tretyakov
> >> > www.griddynamics.com
> >> > itretyakov@griddynamics.com
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Great! That is what I need to know. I was expecting that fsimage can be in
inconsistent state while applying changes and it can be wrong to just copy
it in such moments.
On Tue, Feb 4, 2014 at 11:42 AM, Harsh J <ha...@cloudera.com> wrote:
> Nothing wrong in copying it - except it would come with a bunch of
> redundant information (extra retained edit logs already merged into
> one of the recent fsimages).
>
> On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Thank you Harsh!
> > But I would like to know if just copying NN data directory (dfs.name.dir,
> > dfs.namenode.name.dir) is appropriate or not?
> >
> >
> > On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> You'll ideally just need the latest
> >> fsimage file, which includes the whole check pointed namespace, so
> >> "hdfs dfsadmin -fetch" is appropriate.
> >>
> >> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> >> <it...@griddynamics.com> wrote:
> >> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> >> > dfsadmin
> >> > -fetch', using NN HA configuration?
> >> >
> >> >
> >> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> >> > <it...@griddynamics.com> wrote:
> >> >>
> >> >> Hi!
> >> >>
> >> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >> >>
> >> >> HDFS HA configuration is applied.
> >> >>
> >> >> I would like to ask if there are any difference or possible side
> >> >> effects
> >> >> of backing up namenode data directory.
> >> >>
> >> >> It is ok to just copy NN data directory from active NN to another
> host,
> >> >> what about standby NN?
> >> >> Or it may NN metadate can be in inconsistent state?
> >> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>"
> command?
> >> >>
> >> >> --
> >> >> Best Regards
> >> >> Ivan Tretyakov
> >> >>
> >> >> Deployment Engineer
> >> >> Grid Dynamics
> >> >> +7 812 640 38 76
> >> >> Skype: ivan.v.tretyakov
> >> >> www.griddynamics.com
> >> >> itretyakov@griddynamics.com
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> > Ivan Tretyakov
> >> >
> >> > Deployment Engineer
> >> > Grid Dynamics
> >> > +7 812 640 38 76
> >> > Skype: ivan.v.tretyakov
> >> > www.griddynamics.com
> >> > itretyakov@griddynamics.com
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Great! That is what I need to know. I was expecting that fsimage can be in
inconsistent state while applying changes and it can be wrong to just copy
it in such moments.
On Tue, Feb 4, 2014 at 11:42 AM, Harsh J <ha...@cloudera.com> wrote:
> Nothing wrong in copying it - except it would come with a bunch of
> redundant information (extra retained edit logs already merged into
> one of the recent fsimages).
>
> On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Thank you Harsh!
> > But I would like to know if just copying NN data directory (dfs.name.dir,
> > dfs.namenode.name.dir) is appropriate or not?
> >
> >
> > On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> You'll ideally just need the latest
> >> fsimage file, which includes the whole check pointed namespace, so
> >> "hdfs dfsadmin -fetch" is appropriate.
> >>
> >> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> >> <it...@griddynamics.com> wrote:
> >> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> >> > dfsadmin
> >> > -fetch', using NN HA configuration?
> >> >
> >> >
> >> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> >> > <it...@griddynamics.com> wrote:
> >> >>
> >> >> Hi!
> >> >>
> >> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >> >>
> >> >> HDFS HA configuration is applied.
> >> >>
> >> >> I would like to ask if there are any difference or possible side
> >> >> effects
> >> >> of backing up namenode data directory.
> >> >>
> >> >> It is ok to just copy NN data directory from active NN to another
> host,
> >> >> what about standby NN?
> >> >> Or it may NN metadate can be in inconsistent state?
> >> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>"
> command?
> >> >>
> >> >> --
> >> >> Best Regards
> >> >> Ivan Tretyakov
> >> >>
> >> >> Deployment Engineer
> >> >> Grid Dynamics
> >> >> +7 812 640 38 76
> >> >> Skype: ivan.v.tretyakov
> >> >> www.griddynamics.com
> >> >> itretyakov@griddynamics.com
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> > Ivan Tretyakov
> >> >
> >> > Deployment Engineer
> >> > Grid Dynamics
> >> > +7 812 640 38 76
> >> > Skype: ivan.v.tretyakov
> >> > www.griddynamics.com
> >> > itretyakov@griddynamics.com
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Great! That is what I need to know. I was expecting that fsimage can be in
inconsistent state while applying changes and it can be wrong to just copy
it in such moments.
On Tue, Feb 4, 2014 at 11:42 AM, Harsh J <ha...@cloudera.com> wrote:
> Nothing wrong in copying it - except it would come with a bunch of
> redundant information (extra retained edit logs already merged into
> one of the recent fsimages).
>
> On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Thank you Harsh!
> > But I would like to know if just copying NN data directory (dfs.name.dir,
> > dfs.namenode.name.dir) is appropriate or not?
> >
> >
> > On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi,
> >>
> >> You'll ideally just need the latest
> >> fsimage file, which includes the whole check pointed namespace, so
> >> "hdfs dfsadmin -fetch" is appropriate.
> >>
> >> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> >> <it...@griddynamics.com> wrote:
> >> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> >> > dfsadmin
> >> > -fetch', using NN HA configuration?
> >> >
> >> >
> >> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> >> > <it...@griddynamics.com> wrote:
> >> >>
> >> >> Hi!
> >> >>
> >> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >> >>
> >> >> HDFS HA configuration is applied.
> >> >>
> >> >> I would like to ask if there are any difference or possible side
> >> >> effects
> >> >> of backing up namenode data directory.
> >> >>
> >> >> It is ok to just copy NN data directory from active NN to another
> host,
> >> >> what about standby NN?
> >> >> Or it may NN metadate can be in inconsistent state?
> >> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>"
> command?
> >> >>
> >> >> --
> >> >> Best Regards
> >> >> Ivan Tretyakov
> >> >>
> >> >> Deployment Engineer
> >> >> Grid Dynamics
> >> >> +7 812 640 38 76
> >> >> Skype: ivan.v.tretyakov
> >> >> www.griddynamics.com
> >> >> itretyakov@griddynamics.com
> >> >
> >> >
> >> >
> >> >
> >> > --
> >> > Best Regards
> >> > Ivan Tretyakov
> >> >
> >> > Deployment Engineer
> >> > Grid Dynamics
> >> > +7 812 640 38 76
> >> > Skype: ivan.v.tretyakov
> >> > www.griddynamics.com
> >> > itretyakov@griddynamics.com
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Nothing wrong in copying it - except it would come with a bunch of
redundant information (extra retained edit logs already merged into
one of the recent fsimages).
On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Thank you Harsh!
> But I would like to know if just copying NN data directory (dfs.name.dir,
> dfs.namenode.name.dir) is appropriate or not?
>
>
> On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi,
>>
>> You'll ideally just need the latest
>> fsimage file, which includes the whole check pointed namespace, so
>> "hdfs dfsadmin -fetch" is appropriate.
>>
>> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
>> <it...@griddynamics.com> wrote:
>> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
>> > dfsadmin
>> > -fetch', using NN HA configuration?
>> >
>> >
>> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
>> > <it...@griddynamics.com> wrote:
>> >>
>> >> Hi!
>> >>
>> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>> >>
>> >> HDFS HA configuration is applied.
>> >>
>> >> I would like to ask if there are any difference or possible side
>> >> effects
>> >> of backing up namenode data directory.
>> >>
>> >> It is ok to just copy NN data directory from active NN to another host,
>> >> what about standby NN?
>> >> Or it may NN metadate can be in inconsistent state?
>> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>> >>
>> >> --
>> >> Best Regards
>> >> Ivan Tretyakov
>> >>
>> >> Deployment Engineer
>> >> Grid Dynamics
>> >> +7 812 640 38 76
>> >> Skype: ivan.v.tretyakov
>> >> www.griddynamics.com
>> >> itretyakov@griddynamics.com
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.v.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Nothing wrong in copying it - except it would come with a bunch of
redundant information (extra retained edit logs already merged into
one of the recent fsimages).
On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Thank you Harsh!
> But I would like to know if just copying NN data directory (dfs.name.dir,
> dfs.namenode.name.dir) is appropriate or not?
>
>
> On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi,
>>
>> You'll ideally just need the latest
>> fsimage file, which includes the whole check pointed namespace, so
>> "hdfs dfsadmin -fetch" is appropriate.
>>
>> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
>> <it...@griddynamics.com> wrote:
>> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
>> > dfsadmin
>> > -fetch', using NN HA configuration?
>> >
>> >
>> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
>> > <it...@griddynamics.com> wrote:
>> >>
>> >> Hi!
>> >>
>> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>> >>
>> >> HDFS HA configuration is applied.
>> >>
>> >> I would like to ask if there are any difference or possible side
>> >> effects
>> >> of backing up namenode data directory.
>> >>
>> >> It is ok to just copy NN data directory from active NN to another host,
>> >> what about standby NN?
>> >> Or it may NN metadate can be in inconsistent state?
>> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>> >>
>> >> --
>> >> Best Regards
>> >> Ivan Tretyakov
>> >>
>> >> Deployment Engineer
>> >> Grid Dynamics
>> >> +7 812 640 38 76
>> >> Skype: ivan.v.tretyakov
>> >> www.griddynamics.com
>> >> itretyakov@griddynamics.com
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.v.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Nothing wrong in copying it - except it would come with a bunch of
redundant information (extra retained edit logs already merged into
one of the recent fsimages).
On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Thank you Harsh!
> But I would like to know if just copying NN data directory (dfs.name.dir,
> dfs.namenode.name.dir) is appropriate or not?
>
>
> On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi,
>>
>> You'll ideally just need the latest
>> fsimage file, which includes the whole check pointed namespace, so
>> "hdfs dfsadmin -fetch" is appropriate.
>>
>> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
>> <it...@griddynamics.com> wrote:
>> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
>> > dfsadmin
>> > -fetch', using NN HA configuration?
>> >
>> >
>> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
>> > <it...@griddynamics.com> wrote:
>> >>
>> >> Hi!
>> >>
>> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>> >>
>> >> HDFS HA configuration is applied.
>> >>
>> >> I would like to ask if there are any difference or possible side
>> >> effects
>> >> of backing up namenode data directory.
>> >>
>> >> It is ok to just copy NN data directory from active NN to another host,
>> >> what about standby NN?
>> >> Or it may NN metadate can be in inconsistent state?
>> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>> >>
>> >> --
>> >> Best Regards
>> >> Ivan Tretyakov
>> >>
>> >> Deployment Engineer
>> >> Grid Dynamics
>> >> +7 812 640 38 76
>> >> Skype: ivan.v.tretyakov
>> >> www.griddynamics.com
>> >> itretyakov@griddynamics.com
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.v.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Nothing wrong in copying it - except it would come with a bunch of
redundant information (extra retained edit logs already merged into
one of the recent fsimages).
On Tue, Feb 4, 2014 at 12:07 PM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Thank you Harsh!
> But I would like to know if just copying NN data directory (dfs.name.dir,
> dfs.namenode.name.dir) is appropriate or not?
>
>
> On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi,
>>
>> You'll ideally just need the latest
>> fsimage file, which includes the whole check pointed namespace, so
>> "hdfs dfsadmin -fetch" is appropriate.
>>
>> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
>> <it...@griddynamics.com> wrote:
>> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
>> > dfsadmin
>> > -fetch', using NN HA configuration?
>> >
>> >
>> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
>> > <it...@griddynamics.com> wrote:
>> >>
>> >> Hi!
>> >>
>> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>> >>
>> >> HDFS HA configuration is applied.
>> >>
>> >> I would like to ask if there are any difference or possible side
>> >> effects
>> >> of backing up namenode data directory.
>> >>
>> >> It is ok to just copy NN data directory from active NN to another host,
>> >> what about standby NN?
>> >> Or it may NN metadate can be in inconsistent state?
>> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>> >>
>> >> --
>> >> Best Regards
>> >> Ivan Tretyakov
>> >>
>> >> Deployment Engineer
>> >> Grid Dynamics
>> >> +7 812 640 38 76
>> >> Skype: ivan.v.tretyakov
>> >> www.griddynamics.com
>> >> itretyakov@griddynamics.com
>> >
>> >
>> >
>> >
>> > --
>> > Best Regards
>> > Ivan Tretyakov
>> >
>> > Deployment Engineer
>> > Grid Dynamics
>> > +7 812 640 38 76
>> > Skype: ivan.v.tretyakov
>> > www.griddynamics.com
>> > itretyakov@griddynamics.com
>>
>>
>>
>> --
>> Harsh J
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you Harsh!
But I would like to know if just copying NN data directory (dfs.name.dir,
dfs.namenode.name.dir) is appropriate or not?
On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> You'll ideally just need the latest
> fsimage file, which includes the whole check pointed namespace, so
> "hdfs dfsadmin -fetch" is appropriate.
>
> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> dfsadmin
> > -fetch', using NN HA configuration?
> >
> >
> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> > <it...@griddynamics.com> wrote:
> >>
> >> Hi!
> >>
> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >>
> >> HDFS HA configuration is applied.
> >>
> >> I would like to ask if there are any difference or possible side effects
> >> of backing up namenode data directory.
> >>
> >> It is ok to just copy NN data directory from active NN to another host,
> >> what about standby NN?
> >> Or it may NN metadate can be in inconsistent state?
> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
> >>
> >> --
> >> Best Regards
> >> Ivan Tretyakov
> >>
> >> Deployment Engineer
> >> Grid Dynamics
> >> +7 812 640 38 76
> >> Skype: ivan.v.tretyakov
> >> www.griddynamics.com
> >> itretyakov@griddynamics.com
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you Harsh!
But I would like to know if just copying NN data directory (dfs.name.dir,
dfs.namenode.name.dir) is appropriate or not?
On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> You'll ideally just need the latest
> fsimage file, which includes the whole check pointed namespace, so
> "hdfs dfsadmin -fetch" is appropriate.
>
> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> dfsadmin
> > -fetch', using NN HA configuration?
> >
> >
> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> > <it...@griddynamics.com> wrote:
> >>
> >> Hi!
> >>
> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >>
> >> HDFS HA configuration is applied.
> >>
> >> I would like to ask if there are any difference or possible side effects
> >> of backing up namenode data directory.
> >>
> >> It is ok to just copy NN data directory from active NN to another host,
> >> what about standby NN?
> >> Or it may NN metadate can be in inconsistent state?
> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
> >>
> >> --
> >> Best Regards
> >> Ivan Tretyakov
> >>
> >> Deployment Engineer
> >> Grid Dynamics
> >> +7 812 640 38 76
> >> Skype: ivan.v.tretyakov
> >> www.griddynamics.com
> >> itretyakov@griddynamics.com
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you Harsh!
But I would like to know if just copying NN data directory (dfs.name.dir,
dfs.namenode.name.dir) is appropriate or not?
On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> You'll ideally just need the latest
> fsimage file, which includes the whole check pointed namespace, so
> "hdfs dfsadmin -fetch" is appropriate.
>
> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> dfsadmin
> > -fetch', using NN HA configuration?
> >
> >
> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> > <it...@griddynamics.com> wrote:
> >>
> >> Hi!
> >>
> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >>
> >> HDFS HA configuration is applied.
> >>
> >> I would like to ask if there are any difference or possible side effects
> >> of backing up namenode data directory.
> >>
> >> It is ok to just copy NN data directory from active NN to another host,
> >> what about standby NN?
> >> Or it may NN metadate can be in inconsistent state?
> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
> >>
> >> --
> >> Best Regards
> >> Ivan Tretyakov
> >>
> >> Deployment Engineer
> >> Grid Dynamics
> >> +7 812 640 38 76
> >> Skype: ivan.v.tretyakov
> >> www.griddynamics.com
> >> itretyakov@griddynamics.com
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Thank you Harsh!
But I would like to know if just copying NN data directory (dfs.name.dir,
dfs.namenode.name.dir) is appropriate or not?
On Tue, Feb 4, 2014 at 6:00 AM, Harsh J <ha...@cloudera.com> wrote:
> Hi,
>
> You'll ideally just need the latest
> fsimage file, which includes the whole check pointed namespace, so
> "hdfs dfsadmin -fetch" is appropriate.
>
> On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
> > Does anybody can comment 'copying NN metadata directory' vs 'hdfs
> dfsadmin
> > -fetch', using NN HA configuration?
> >
> >
> > On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> > <it...@griddynamics.com> wrote:
> >>
> >> Hi!
> >>
> >> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
> >>
> >> HDFS HA configuration is applied.
> >>
> >> I would like to ask if there are any difference or possible side effects
> >> of backing up namenode data directory.
> >>
> >> It is ok to just copy NN data directory from active NN to another host,
> >> what about standby NN?
> >> Or it may NN metadate can be in inconsistent state?
> >> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
> >>
> >> --
> >> Best Regards
> >> Ivan Tretyakov
> >>
> >> Deployment Engineer
> >> Grid Dynamics
> >> +7 812 640 38 76
> >> Skype: ivan.v.tretyakov
> >> www.griddynamics.com
> >> itretyakov@griddynamics.com
> >
> >
> >
> >
> > --
> > Best Regards
> > Ivan Tretyakov
> >
> > Deployment Engineer
> > Grid Dynamics
> > +7 812 640 38 76
> > Skype: ivan.v.tretyakov
> > www.griddynamics.com
> > itretyakov@griddynamics.com
>
>
>
> --
> Harsh J
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi,
You'll ideally just need the latest
fsimage file, which includes the whole check pointed namespace, so
"hdfs dfsadmin -fetch" is appropriate.
On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
> -fetch', using NN HA configuration?
>
>
> On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
>>
>> Hi!
>>
>> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>>
>> HDFS HA configuration is applied.
>>
>> I would like to ask if there are any difference or possible side effects
>> of backing up namenode data directory.
>>
>> It is ok to just copy NN data directory from active NN to another host,
>> what about standby NN?
>> Or it may NN metadate can be in inconsistent state?
>> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.v.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi,
You'll ideally just need the latest
fsimage file, which includes the whole check pointed namespace, so
"hdfs dfsadmin -fetch" is appropriate.
On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
> -fetch', using NN HA configuration?
>
>
> On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
>>
>> Hi!
>>
>> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>>
>> HDFS HA configuration is applied.
>>
>> I would like to ask if there are any difference or possible side effects
>> of backing up namenode data directory.
>>
>> It is ok to just copy NN data directory from active NN to another host,
>> what about standby NN?
>> Or it may NN metadate can be in inconsistent state?
>> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.v.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi,
You'll ideally just need the latest
fsimage file, which includes the whole check pointed namespace, so
"hdfs dfsadmin -fetch" is appropriate.
On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
> -fetch', using NN HA configuration?
>
>
> On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
>>
>> Hi!
>>
>> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>>
>> HDFS HA configuration is applied.
>>
>> I would like to ask if there are any difference or possible side effects
>> of backing up namenode data directory.
>>
>> It is ok to just copy NN data directory from active NN to another host,
>> what about standby NN?
>> Or it may NN metadate can be in inconsistent state?
>> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.v.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Harsh J <ha...@cloudera.com>.
Hi,
You'll ideally just need the latest
fsimage file, which includes the whole check pointed namespace, so
"hdfs dfsadmin -fetch" is appropriate.
On Tue, Feb 4, 2014 at 3:02 AM, Ivan Tretyakov
<it...@griddynamics.com> wrote:
> Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
> -fetch', using NN HA configuration?
>
>
> On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov
> <it...@griddynamics.com> wrote:
>>
>> Hi!
>>
>> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>>
>> HDFS HA configuration is applied.
>>
>> I would like to ask if there are any difference or possible side effects
>> of backing up namenode data directory.
>>
>> It is ok to just copy NN data directory from active NN to another host,
>> what about standby NN?
>> Or it may NN metadate can be in inconsistent state?
>> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>>
>> --
>> Best Regards
>> Ivan Tretyakov
>>
>> Deployment Engineer
>> Grid Dynamics
>> +7 812 640 38 76
>> Skype: ivan.v.tretyakov
>> www.griddynamics.com
>> itretyakov@griddynamics.com
>
>
>
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
--
Harsh J
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
-fetch', using NN HA configuration?
On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hi!
>
> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>
> HDFS HA configuration is applied.
>
> I would like to ask if there are any difference or possible side effects
> of backing up namenode data directory.
>
> It is ok to just copy NN data directory from active NN to another host,
> what about standby NN?
> Or it may NN metadate can be in inconsistent state?
> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
-fetch', using NN HA configuration?
On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hi!
>
> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>
> HDFS HA configuration is applied.
>
> I would like to ask if there are any difference or possible side effects
> of backing up namenode data directory.
>
> It is ok to just copy NN data directory from active NN to another host,
> what about standby NN?
> Or it may NN metadate can be in inconsistent state?
> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
-fetch', using NN HA configuration?
On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hi!
>
> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>
> HDFS HA configuration is applied.
>
> I would like to ask if there are any difference or possible side effects
> of backing up namenode data directory.
>
> It is ok to just copy NN data directory from active NN to another host,
> what about standby NN?
> Or it may NN metadate can be in inconsistent state?
> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com
Re: Backup NameNode metadata in HA configuration
Posted by Ivan Tretyakov <it...@griddynamics.com>.
Does anybody can comment 'copying NN metadata directory' vs 'hdfs dfsadmin
-fetch', using NN HA configuration?
On Fri, Jan 31, 2014 at 8:26 PM, Ivan Tretyakov <itretyakov@griddynamics.com
> wrote:
> Hi!
>
> We are using following version of Hadoop software: 2.0.0-cdh4.1.1.
>
> HDFS HA configuration is applied.
>
> I would like to ask if there are any difference or possible side effects
> of backing up namenode data directory.
>
> It is ok to just copy NN data directory from active NN to another host,
> what about standby NN?
> Or it may NN metadate can be in inconsistent state?
> Is there any advantages of using "hdfs dfsadmin -fetch <file>" command?
>
> --
> Best Regards
> Ivan Tretyakov
>
> Deployment Engineer
> Grid Dynamics
> +7 812 640 38 76
> Skype: ivan.v.tretyakov
> www.griddynamics.com
> itretyakov@griddynamics.com
>
--
Best Regards
Ivan Tretyakov
Deployment Engineer
Grid Dynamics
+7 812 640 38 76
Skype: ivan.v.tretyakov
www.griddynamics.com
itretyakov@griddynamics.com