backup node question

[Lin Ma <li...@gmail.com>]

Hi guys,

I am reading from this paper to learn about backup nodes (
http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),

   - It is mentioned, "It contains all file system metadata information
   except for block locations. It can perform all operations of the regular
   NameNode that do not involve modification of the namespace or knowledge of
   block locations. ", what kinds of operations do not need knowledge of block
   locations?
   - It is also mentioned, "Use of a BackupNode provides the option of
   running the NameNode without persistent storage, delegating responsibility
   for the namespace state persisting to the BackupNode.", what means "running
   the NameNode without persistent storage" and "delegating responsibility for
   the namespace state persisting"?

regards,

Lin

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

oh, got it. you are a good guy.

--Send from my Sony mobile.
On Apr 7, 2013 10:11 PM, "Harsh J" <ha...@cloudera.com> wrote:

> StandbyNameNode is the term we use to refer to a NameNode in HA that
> is currently not the active one (i.e. its state is 'Standby'). Its not
> a special type of daemon (i.e. it just runs the NameNode service),
> just a naming convention.
>
> On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> > I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> > Standby Namenode?
> >
> > --Send from my Sony mobile.
> >
> > On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
> >>
> >> BackupNameNode is not present in the maintenance 1.x releases, it is a
> >> feature added to a higher version; you can try it out in 2.x today if
> >> you wish to.
> >>
> >> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> >> > Hi Harsh,
> >> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> >> those are passed synchronously to the BackupNameNode, which persists
> >> >> it on its behalf.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Thanks Harsh,
> >> >> >
> >> >> > For your comments, "What it means is that the NameNode need not
> store
> >> >> > anything locally", you mean Primary Name Node do not need to store
> >> >> > checkpoint/journal locally, and only need to keep memory image
> >> >> > up-to-date
> >> >> > for edits?
> >> >> >
> >> >> > regards,
> >> >> > Lin
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com>
> wrote:
> >> >> >>
> >> >> >> Hi Lin,
> >> >> >>
> >> >> >> My reply inline.
> >> >> >>
> >> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> >> > Hi guys,
> >> >> >> >
> >> >> >> > I am reading from this paper to learn about backup nodes
> >> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf
> ),
> >> >> >> >
> >> >> >> > It is mentioned, "It contains all file system metadata
> information
> >> >> >> > except
> >> >> >> > for block locations. It can perform all operations of the
> regular
> >> >> >> > NameNode
> >> >> >> > that do not involve modification of the namespace or knowledge
> of
> >> >> >> > block
> >> >> >> > locations. ", what kinds of operations do not need knowledge of
> >> >> >> > block
> >> >> >> > locations?
> >> >> >>
> >> >> >> Operations that do not involve data reads or writes would not
> >> >> >> require
> >> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> >> namespace mutation, an example would be listing directories and
> >> >> >> looking up file information via FileStatus objects (perhaps the
> only
> >> >> >> examples - its like a safemode but no reads either).
> >> >> >>
> >> >> >> > It is also mentioned, "Use of a BackupNode provides the option
> of
> >> >> >> > running
> >> >> >> > the NameNode without persistent storage, delegating
> responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting to the BackupNode.", what means
> >> >> >> > "running
> >> >> >> > the
> >> >> >> > NameNode without persistent storage" and "delegating
> >> >> >> > responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting"?
> >> >> >>
> >> >> >> What it means is that the NameNode need not store anything
> locally,
> >> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> >> current checkpoint from the BNN and boot up anywhere, since
> there's
> >> >> >> no
> >> >> >> local storage requirement.
> >> >> >>
> >> >> >> --
> >> >> >> Harsh J
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

oh, got it. you are a good guy.

--Send from my Sony mobile.
On Apr 7, 2013 10:11 PM, "Harsh J" <ha...@cloudera.com> wrote:

> StandbyNameNode is the term we use to refer to a NameNode in HA that
> is currently not the active one (i.e. its state is 'Standby'). Its not
> a special type of daemon (i.e. it just runs the NameNode service),
> just a naming convention.
>
> On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> > I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> > Standby Namenode?
> >
> > --Send from my Sony mobile.
> >
> > On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
> >>
> >> BackupNameNode is not present in the maintenance 1.x releases, it is a
> >> feature added to a higher version; you can try it out in 2.x today if
> >> you wish to.
> >>
> >> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> >> > Hi Harsh,
> >> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> >> those are passed synchronously to the BackupNameNode, which persists
> >> >> it on its behalf.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Thanks Harsh,
> >> >> >
> >> >> > For your comments, "What it means is that the NameNode need not
> store
> >> >> > anything locally", you mean Primary Name Node do not need to store
> >> >> > checkpoint/journal locally, and only need to keep memory image
> >> >> > up-to-date
> >> >> > for edits?
> >> >> >
> >> >> > regards,
> >> >> > Lin
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com>
> wrote:
> >> >> >>
> >> >> >> Hi Lin,
> >> >> >>
> >> >> >> My reply inline.
> >> >> >>
> >> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> >> > Hi guys,
> >> >> >> >
> >> >> >> > I am reading from this paper to learn about backup nodes
> >> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf
> ),
> >> >> >> >
> >> >> >> > It is mentioned, "It contains all file system metadata
> information
> >> >> >> > except
> >> >> >> > for block locations. It can perform all operations of the
> regular
> >> >> >> > NameNode
> >> >> >> > that do not involve modification of the namespace or knowledge
> of
> >> >> >> > block
> >> >> >> > locations. ", what kinds of operations do not need knowledge of
> >> >> >> > block
> >> >> >> > locations?
> >> >> >>
> >> >> >> Operations that do not involve data reads or writes would not
> >> >> >> require
> >> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> >> namespace mutation, an example would be listing directories and
> >> >> >> looking up file information via FileStatus objects (perhaps the
> only
> >> >> >> examples - its like a safemode but no reads either).
> >> >> >>
> >> >> >> > It is also mentioned, "Use of a BackupNode provides the option
> of
> >> >> >> > running
> >> >> >> > the NameNode without persistent storage, delegating
> responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting to the BackupNode.", what means
> >> >> >> > "running
> >> >> >> > the
> >> >> >> > NameNode without persistent storage" and "delegating
> >> >> >> > responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting"?
> >> >> >>
> >> >> >> What it means is that the NameNode need not store anything
> locally,
> >> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> >> current checkpoint from the BNN and boot up anywhere, since
> there's
> >> >> >> no
> >> >> >> local storage requirement.
> >> >> >>
> >> >> >> --
> >> >> >> Harsh J
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

oh, got it. you are a good guy.

--Send from my Sony mobile.
On Apr 7, 2013 10:11 PM, "Harsh J" <ha...@cloudera.com> wrote:

> StandbyNameNode is the term we use to refer to a NameNode in HA that
> is currently not the active one (i.e. its state is 'Standby'). Its not
> a special type of daemon (i.e. it just runs the NameNode service),
> just a naming convention.
>
> On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> > I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> > Standby Namenode?
> >
> > --Send from my Sony mobile.
> >
> > On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
> >>
> >> BackupNameNode is not present in the maintenance 1.x releases, it is a
> >> feature added to a higher version; you can try it out in 2.x today if
> >> you wish to.
> >>
> >> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> >> > Hi Harsh,
> >> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> >> those are passed synchronously to the BackupNameNode, which persists
> >> >> it on its behalf.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Thanks Harsh,
> >> >> >
> >> >> > For your comments, "What it means is that the NameNode need not
> store
> >> >> > anything locally", you mean Primary Name Node do not need to store
> >> >> > checkpoint/journal locally, and only need to keep memory image
> >> >> > up-to-date
> >> >> > for edits?
> >> >> >
> >> >> > regards,
> >> >> > Lin
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com>
> wrote:
> >> >> >>
> >> >> >> Hi Lin,
> >> >> >>
> >> >> >> My reply inline.
> >> >> >>
> >> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> >> > Hi guys,
> >> >> >> >
> >> >> >> > I am reading from this paper to learn about backup nodes
> >> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf
> ),
> >> >> >> >
> >> >> >> > It is mentioned, "It contains all file system metadata
> information
> >> >> >> > except
> >> >> >> > for block locations. It can perform all operations of the
> regular
> >> >> >> > NameNode
> >> >> >> > that do not involve modification of the namespace or knowledge
> of
> >> >> >> > block
> >> >> >> > locations. ", what kinds of operations do not need knowledge of
> >> >> >> > block
> >> >> >> > locations?
> >> >> >>
> >> >> >> Operations that do not involve data reads or writes would not
> >> >> >> require
> >> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> >> namespace mutation, an example would be listing directories and
> >> >> >> looking up file information via FileStatus objects (perhaps the
> only
> >> >> >> examples - its like a safemode but no reads either).
> >> >> >>
> >> >> >> > It is also mentioned, "Use of a BackupNode provides the option
> of
> >> >> >> > running
> >> >> >> > the NameNode without persistent storage, delegating
> responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting to the BackupNode.", what means
> >> >> >> > "running
> >> >> >> > the
> >> >> >> > NameNode without persistent storage" and "delegating
> >> >> >> > responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting"?
> >> >> >>
> >> >> >> What it means is that the NameNode need not store anything
> locally,
> >> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> >> current checkpoint from the BNN and boot up anywhere, since
> there's
> >> >> >> no
> >> >> >> local storage requirement.
> >> >> >>
> >> >> >> --
> >> >> >> Harsh J
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

oh, got it. you are a good guy.

--Send from my Sony mobile.
On Apr 7, 2013 10:11 PM, "Harsh J" <ha...@cloudera.com> wrote:

> StandbyNameNode is the term we use to refer to a NameNode in HA that
> is currently not the active one (i.e. its state is 'Standby'). Its not
> a special type of daemon (i.e. it just runs the NameNode service),
> just a naming convention.
>
> On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> > I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> > Standby Namenode?
> >
> > --Send from my Sony mobile.
> >
> > On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
> >>
> >> BackupNameNode is not present in the maintenance 1.x releases, it is a
> >> feature added to a higher version; you can try it out in 2.x today if
> >> you wish to.
> >>
> >> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> >> > Hi Harsh,
> >> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> >> those are passed synchronously to the BackupNameNode, which persists
> >> >> it on its behalf.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Thanks Harsh,
> >> >> >
> >> >> > For your comments, "What it means is that the NameNode need not
> store
> >> >> > anything locally", you mean Primary Name Node do not need to store
> >> >> > checkpoint/journal locally, and only need to keep memory image
> >> >> > up-to-date
> >> >> > for edits?
> >> >> >
> >> >> > regards,
> >> >> > Lin
> >> >> >
> >> >> >
> >> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com>
> wrote:
> >> >> >>
> >> >> >> Hi Lin,
> >> >> >>
> >> >> >> My reply inline.
> >> >> >>
> >> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> >> > Hi guys,
> >> >> >> >
> >> >> >> > I am reading from this paper to learn about backup nodes
> >> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf
> ),
> >> >> >> >
> >> >> >> > It is mentioned, "It contains all file system metadata
> information
> >> >> >> > except
> >> >> >> > for block locations. It can perform all operations of the
> regular
> >> >> >> > NameNode
> >> >> >> > that do not involve modification of the namespace or knowledge
> of
> >> >> >> > block
> >> >> >> > locations. ", what kinds of operations do not need knowledge of
> >> >> >> > block
> >> >> >> > locations?
> >> >> >>
> >> >> >> Operations that do not involve data reads or writes would not
> >> >> >> require
> >> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> >> namespace mutation, an example would be listing directories and
> >> >> >> looking up file information via FileStatus objects (perhaps the
> only
> >> >> >> examples - its like a safemode but no reads either).
> >> >> >>
> >> >> >> > It is also mentioned, "Use of a BackupNode provides the option
> of
> >> >> >> > running
> >> >> >> > the NameNode without persistent storage, delegating
> responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting to the BackupNode.", what means
> >> >> >> > "running
> >> >> >> > the
> >> >> >> > NameNode without persistent storage" and "delegating
> >> >> >> > responsibility
> >> >> >> > for
> >> >> >> > the
> >> >> >> > namespace state persisting"?
> >> >> >>
> >> >> >> What it means is that the NameNode need not store anything
> locally,
> >> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> >> current checkpoint from the BNN and boot up anywhere, since
> there's
> >> >> >> no
> >> >> >> local storage requirement.
> >> >> >>
> >> >> >> --
> >> >> >> Harsh J
> >> >> >
> >> >> >
> >> >>
> >> >>
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
>
>
>
> --
> Harsh J
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

StandbyNameNode is the term we use to refer to a NameNode in HA that
is currently not the active one (i.e. its state is 'Standby'). Its not
a special type of daemon (i.e. it just runs the NameNode service),
just a naming convention.

On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
>
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> >> >> > block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> >> >> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> >> >> > "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> >> >> > responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> >> >> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J



-- 
Harsh J

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

SNN=secondary name node  in my last mail.

--Send from my Sony mobile.
On Apr 7, 2013 10:01 PM, "Azuryy Yu" <az...@gmail.com> wrote:

> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

StandbyNameNode is the term we use to refer to a NameNode in HA that
is currently not the active one (i.e. its state is 'Standby'). Its not
a special type of daemon (i.e. it just runs the NameNode service),
just a naming convention.

On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
>
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> >> >> > block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> >> >> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> >> >> > "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> >> >> > responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> >> >> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

StandbyNameNode is the term we use to refer to a NameNode in HA that
is currently not the active one (i.e. its state is 'Standby'). Its not
a special type of daemon (i.e. it just runs the NameNode service),
just a naming convention.

On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
>
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> >> >> > block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> >> >> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> >> >> > "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> >> >> > responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> >> >> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J



-- 
Harsh J

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

SNN=secondary name node  in my last mail.

--Send from my Sony mobile.
On Apr 7, 2013 10:01 PM, "Azuryy Yu" <az...@gmail.com> wrote:

> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

SNN=secondary name node  in my last mail.

--Send from my Sony mobile.
On Apr 7, 2013 10:01 PM, "Azuryy Yu" <az...@gmail.com> wrote:

> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

StandbyNameNode is the term we use to refer to a NameNode in HA that
is currently not the active one (i.e. its state is 'Standby'). Its not
a special type of daemon (i.e. it just runs the NameNode service),
just a naming convention.

On Sun, Apr 7, 2013 at 7:31 PM, Azuryy Yu <az...@gmail.com> wrote:
> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
>
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> >> >> > block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> >> >> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> >> >> > "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> >> >> > responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> >> >> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J



-- 
Harsh J

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

SNN=secondary name node  in my last mail.

--Send from my Sony mobile.
On Apr 7, 2013 10:01 PM, "Azuryy Yu" <az...@gmail.com> wrote:

> I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
> Standby Namenode?
>
> --Send from my Sony mobile.
> On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:
>
>> BackupNameNode is not present in the maintenance 1.x releases, it is a
>> feature added to a higher version; you can try it out in 2.x today if
>> you wish to.
>>
>> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
>> > Hi Harsh,
>> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>> >
>> >
>> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Yes, it need not keep an edits (transactions) stream locally cause
>> >> those are passed synchronously to the BackupNameNode, which persists
>> >> it on its behalf.
>> >>
>> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Thanks Harsh,
>> >> >
>> >> > For your comments, "What it means is that the NameNode need not store
>> >> > anything locally", you mean Primary Name Node do not need to store
>> >> > checkpoint/journal locally, and only need to keep memory image
>> >> > up-to-date
>> >> > for edits?
>> >> >
>> >> > regards,
>> >> > Lin
>> >> >
>> >> >
>> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >> >>
>> >> >> Hi Lin,
>> >> >>
>> >> >> My reply inline.
>> >> >>
>> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> >> > Hi guys,
>> >> >> >
>> >> >> > I am reading from this paper to learn about backup nodes
>> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >> >
>> >> >> > It is mentioned, "It contains all file system metadata information
>> >> >> > except
>> >> >> > for block locations. It can perform all operations of the regular
>> >> >> > NameNode
>> >> >> > that do not involve modification of the namespace or knowledge of
>> >> >> > block
>> >> >> > locations. ", what kinds of operations do not need knowledge of
>> block
>> >> >> > locations?
>> >> >>
>> >> >> Operations that do not involve data reads or writes would not
>> require
>> >> >> knowledge of block locations. Applying also the restriction of no
>> >> >> namespace mutation, an example would be listing directories and
>> >> >> looking up file information via FileStatus objects (perhaps the only
>> >> >> examples - its like a safemode but no reads either).
>> >> >>
>> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> >> > running
>> >> >> > the NameNode without persistent storage, delegating responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting to the BackupNode.", what means
>> "running
>> >> >> > the
>> >> >> > NameNode without persistent storage" and "delegating
>> responsibility
>> >> >> > for
>> >> >> > the
>> >> >> > namespace state persisting"?
>> >> >>
>> >> >> What it means is that the NameNode need not store anything locally,
>> >> >> but can rely on the edits being stored at the BackupNameNode which
>> >> >> would continuously be receiving it. When restarted, it can grab a
>> >> >> current checkpoint from the BNN and boot up anywhere, since there's
>> no
>> >> >> local storage requirement.
>> >> >>
>> >> >> --
>> >> >> Harsh J
>> >> >
>> >> >
>> >>
>> >>
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>>
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
Standby Namenode?

--Send from my Sony mobile.
On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:

> BackupNameNode is not present in the maintenance 1.x releases, it is a
> feature added to a higher version; you can try it out in 2.x today if
> you wish to.
>
> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi Harsh,
> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >
> >
> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> those are passed synchronously to the BackupNameNode, which persists
> >> it on its behalf.
> >>
> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Thanks Harsh,
> >> >
> >> > For your comments, "What it means is that the NameNode need not store
> >> > anything locally", you mean Primary Name Node do not need to store
> >> > checkpoint/journal locally, and only need to keep memory image
> >> > up-to-date
> >> > for edits?
> >> >
> >> > regards,
> >> > Lin
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Hi Lin,
> >> >>
> >> >> My reply inline.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > I am reading from this paper to learn about backup nodes
> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >> >
> >> >> > It is mentioned, "It contains all file system metadata information
> >> >> > except
> >> >> > for block locations. It can perform all operations of the regular
> >> >> > NameNode
> >> >> > that do not involve modification of the namespace or knowledge of
> >> >> > block
> >> >> > locations. ", what kinds of operations do not need knowledge of
> block
> >> >> > locations?
> >> >>
> >> >> Operations that do not involve data reads or writes would not require
> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> namespace mutation, an example would be listing directories and
> >> >> looking up file information via FileStatus objects (perhaps the only
> >> >> examples - its like a safemode but no reads either).
> >> >>
> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> >> > running
> >> >> > the NameNode without persistent storage, delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting to the BackupNode.", what means "running
> >> >> > the
> >> >> > NameNode without persistent storage" and "delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting"?
> >> >>
> >> >> What it means is that the NameNode need not store anything locally,
> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> current checkpoint from the BNN and boot up anywhere, since there's
> no
> >> >> local storage requirement.
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
Standby Namenode?

--Send from my Sony mobile.
On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:

> BackupNameNode is not present in the maintenance 1.x releases, it is a
> feature added to a higher version; you can try it out in 2.x today if
> you wish to.
>
> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi Harsh,
> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >
> >
> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> those are passed synchronously to the BackupNameNode, which persists
> >> it on its behalf.
> >>
> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Thanks Harsh,
> >> >
> >> > For your comments, "What it means is that the NameNode need not store
> >> > anything locally", you mean Primary Name Node do not need to store
> >> > checkpoint/journal locally, and only need to keep memory image
> >> > up-to-date
> >> > for edits?
> >> >
> >> > regards,
> >> > Lin
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Hi Lin,
> >> >>
> >> >> My reply inline.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > I am reading from this paper to learn about backup nodes
> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >> >
> >> >> > It is mentioned, "It contains all file system metadata information
> >> >> > except
> >> >> > for block locations. It can perform all operations of the regular
> >> >> > NameNode
> >> >> > that do not involve modification of the namespace or knowledge of
> >> >> > block
> >> >> > locations. ", what kinds of operations do not need knowledge of
> block
> >> >> > locations?
> >> >>
> >> >> Operations that do not involve data reads or writes would not require
> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> namespace mutation, an example would be listing directories and
> >> >> looking up file information via FileStatus objects (perhaps the only
> >> >> examples - its like a safemode but no reads either).
> >> >>
> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> >> > running
> >> >> > the NameNode without persistent storage, delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting to the BackupNode.", what means "running
> >> >> > the
> >> >> > NameNode without persistent storage" and "delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting"?
> >> >>
> >> >> What it means is that the NameNode need not store anything locally,
> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> current checkpoint from the BNN and boot up anywhere, since there's
> no
> >> >> local storage requirement.
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
Standby Namenode?

--Send from my Sony mobile.
On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:

> BackupNameNode is not present in the maintenance 1.x releases, it is a
> feature added to a higher version; you can try it out in 2.x today if
> you wish to.
>
> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi Harsh,
> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >
> >
> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> those are passed synchronously to the BackupNameNode, which persists
> >> it on its behalf.
> >>
> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Thanks Harsh,
> >> >
> >> > For your comments, "What it means is that the NameNode need not store
> >> > anything locally", you mean Primary Name Node do not need to store
> >> > checkpoint/journal locally, and only need to keep memory image
> >> > up-to-date
> >> > for edits?
> >> >
> >> > regards,
> >> > Lin
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Hi Lin,
> >> >>
> >> >> My reply inline.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > I am reading from this paper to learn about backup nodes
> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >> >
> >> >> > It is mentioned, "It contains all file system metadata information
> >> >> > except
> >> >> > for block locations. It can perform all operations of the regular
> >> >> > NameNode
> >> >> > that do not involve modification of the namespace or knowledge of
> >> >> > block
> >> >> > locations. ", what kinds of operations do not need knowledge of
> block
> >> >> > locations?
> >> >>
> >> >> Operations that do not involve data reads or writes would not require
> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> namespace mutation, an example would be listing directories and
> >> >> looking up file information via FileStatus objects (perhaps the only
> >> >> examples - its like a safemode but no reads either).
> >> >>
> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> >> > running
> >> >> > the NameNode without persistent storage, delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting to the BackupNode.", what means "running
> >> >> > the
> >> >> > NameNode without persistent storage" and "delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting"?
> >> >>
> >> >> What it means is that the NameNode need not store anything locally,
> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> current checkpoint from the BNN and boot up anywhere, since there's
> no
> >> >> local storage requirement.
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

I am confused. Hadoopv2 has NN SNN DN JN(journal node), so whats
Standby Namenode?

--Send from my Sony mobile.
On Apr 7, 2013 9:03 PM, "Harsh J" <ha...@cloudera.com> wrote:

> BackupNameNode is not present in the maintenance 1.x releases, it is a
> feature added to a higher version; you can try it out in 2.x today if
> you wish to.
>
> On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> > Hi Harsh,
> > Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
> >
> >
> > On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Yes, it need not keep an edits (transactions) stream locally cause
> >> those are passed synchronously to the BackupNameNode, which persists
> >> it on its behalf.
> >>
> >> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Thanks Harsh,
> >> >
> >> > For your comments, "What it means is that the NameNode need not store
> >> > anything locally", you mean Primary Name Node do not need to store
> >> > checkpoint/journal locally, and only need to keep memory image
> >> > up-to-date
> >> > for edits?
> >> >
> >> > regards,
> >> > Lin
> >> >
> >> >
> >> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >> >>
> >> >> Hi Lin,
> >> >>
> >> >> My reply inline.
> >> >>
> >> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> >> > Hi guys,
> >> >> >
> >> >> > I am reading from this paper to learn about backup nodes
> >> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >> >
> >> >> > It is mentioned, "It contains all file system metadata information
> >> >> > except
> >> >> > for block locations. It can perform all operations of the regular
> >> >> > NameNode
> >> >> > that do not involve modification of the namespace or knowledge of
> >> >> > block
> >> >> > locations. ", what kinds of operations do not need knowledge of
> block
> >> >> > locations?
> >> >>
> >> >> Operations that do not involve data reads or writes would not require
> >> >> knowledge of block locations. Applying also the restriction of no
> >> >> namespace mutation, an example would be listing directories and
> >> >> looking up file information via FileStatus objects (perhaps the only
> >> >> examples - its like a safemode but no reads either).
> >> >>
> >> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> >> > running
> >> >> > the NameNode without persistent storage, delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting to the BackupNode.", what means "running
> >> >> > the
> >> >> > NameNode without persistent storage" and "delegating responsibility
> >> >> > for
> >> >> > the
> >> >> > namespace state persisting"?
> >> >>
> >> >> What it means is that the NameNode need not store anything locally,
> >> >> but can rely on the edits being stored at the BackupNameNode which
> >> >> would continuously be receiving it. When restarted, it can grab a
> >> >> current checkpoint from the BNN and boot up anywhere, since there's
> no
> >> >> local storage requirement.
> >> >>
> >> >> --
> >> >> Harsh J
> >> >
> >> >
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

BackupNameNode is not present in the maintenance 1.x releases, it is a
feature added to a higher version; you can try it out in 2.x today if
you wish to.

On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Harsh,
> Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>
>
> On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Yes, it need not keep an edits (transactions) stream locally cause
>> those are passed synchronously to the BackupNameNode, which persists
>> it on its behalf.
>>
>> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> > Thanks Harsh,
>> >
>> > For your comments, "What it means is that the NameNode need not store
>> > anything locally", you mean Primary Name Node do not need to store
>> > checkpoint/journal locally, and only need to keep memory image
>> > up-to-date
>> > for edits?
>> >
>> > regards,
>> > Lin
>> >
>> >
>> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Hi Lin,
>> >>
>> >> My reply inline.
>> >>
>> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I am reading from this paper to learn about backup nodes
>> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >
>> >> > It is mentioned, "It contains all file system metadata information
>> >> > except
>> >> > for block locations. It can perform all operations of the regular
>> >> > NameNode
>> >> > that do not involve modification of the namespace or knowledge of
>> >> > block
>> >> > locations. ", what kinds of operations do not need knowledge of block
>> >> > locations?
>> >>
>> >> Operations that do not involve data reads or writes would not require
>> >> knowledge of block locations. Applying also the restriction of no
>> >> namespace mutation, an example would be listing directories and
>> >> looking up file information via FileStatus objects (perhaps the only
>> >> examples - its like a safemode but no reads either).
>> >>
>> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> > running
>> >> > the NameNode without persistent storage, delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting to the BackupNode.", what means "running
>> >> > the
>> >> > NameNode without persistent storage" and "delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting"?
>> >>
>> >> What it means is that the NameNode need not store anything locally,
>> >> but can rely on the edits being stored at the BackupNameNode which
>> >> would continuously be receiving it. When restarted, it can grab a
>> >> current checkpoint from the BNN and boot up anywhere, since there's no
>> >> local storage requirement.
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

BackupNameNode is not present in the maintenance 1.x releases, it is a
feature added to a higher version; you can try it out in 2.x today if
you wish to.

On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Harsh,
> Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>
>
> On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Yes, it need not keep an edits (transactions) stream locally cause
>> those are passed synchronously to the BackupNameNode, which persists
>> it on its behalf.
>>
>> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> > Thanks Harsh,
>> >
>> > For your comments, "What it means is that the NameNode need not store
>> > anything locally", you mean Primary Name Node do not need to store
>> > checkpoint/journal locally, and only need to keep memory image
>> > up-to-date
>> > for edits?
>> >
>> > regards,
>> > Lin
>> >
>> >
>> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Hi Lin,
>> >>
>> >> My reply inline.
>> >>
>> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I am reading from this paper to learn about backup nodes
>> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >
>> >> > It is mentioned, "It contains all file system metadata information
>> >> > except
>> >> > for block locations. It can perform all operations of the regular
>> >> > NameNode
>> >> > that do not involve modification of the namespace or knowledge of
>> >> > block
>> >> > locations. ", what kinds of operations do not need knowledge of block
>> >> > locations?
>> >>
>> >> Operations that do not involve data reads or writes would not require
>> >> knowledge of block locations. Applying also the restriction of no
>> >> namespace mutation, an example would be listing directories and
>> >> looking up file information via FileStatus objects (perhaps the only
>> >> examples - its like a safemode but no reads either).
>> >>
>> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> > running
>> >> > the NameNode without persistent storage, delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting to the BackupNode.", what means "running
>> >> > the
>> >> > NameNode without persistent storage" and "delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting"?
>> >>
>> >> What it means is that the NameNode need not store anything locally,
>> >> but can rely on the edits being stored at the BackupNameNode which
>> >> would continuously be receiving it. When restarted, it can grab a
>> >> current checkpoint from the BNN and boot up anywhere, since there's no
>> >> local storage requirement.
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

BackupNameNode is not present in the maintenance 1.x releases, it is a
feature added to a higher version; you can try it out in 2.x today if
you wish to.

On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Harsh,
> Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>
>
> On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Yes, it need not keep an edits (transactions) stream locally cause
>> those are passed synchronously to the BackupNameNode, which persists
>> it on its behalf.
>>
>> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> > Thanks Harsh,
>> >
>> > For your comments, "What it means is that the NameNode need not store
>> > anything locally", you mean Primary Name Node do not need to store
>> > checkpoint/journal locally, and only need to keep memory image
>> > up-to-date
>> > for edits?
>> >
>> > regards,
>> > Lin
>> >
>> >
>> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Hi Lin,
>> >>
>> >> My reply inline.
>> >>
>> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I am reading from this paper to learn about backup nodes
>> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >
>> >> > It is mentioned, "It contains all file system metadata information
>> >> > except
>> >> > for block locations. It can perform all operations of the regular
>> >> > NameNode
>> >> > that do not involve modification of the namespace or knowledge of
>> >> > block
>> >> > locations. ", what kinds of operations do not need knowledge of block
>> >> > locations?
>> >>
>> >> Operations that do not involve data reads or writes would not require
>> >> knowledge of block locations. Applying also the restriction of no
>> >> namespace mutation, an example would be listing directories and
>> >> looking up file information via FileStatus objects (perhaps the only
>> >> examples - its like a safemode but no reads either).
>> >>
>> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> > running
>> >> > the NameNode without persistent storage, delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting to the BackupNode.", what means "running
>> >> > the
>> >> > NameNode without persistent storage" and "delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting"?
>> >>
>> >> What it means is that the NameNode need not store anything locally,
>> >> but can rely on the edits being stored at the BackupNameNode which
>> >> would continuously be receiving it. When restarted, it can grab a
>> >> current checkpoint from the BNN and boot up anywhere, since there's no
>> >> local storage requirement.
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

BackupNameNode is not present in the maintenance 1.x releases, it is a
feature added to a higher version; you can try it out in 2.x today if
you wish to.

On Sun, Apr 7, 2013 at 3:12 PM, Azuryy Yu <az...@gmail.com> wrote:
> Hi Harsh,
> Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?
>
>
> On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Yes, it need not keep an edits (transactions) stream locally cause
>> those are passed synchronously to the BackupNameNode, which persists
>> it on its behalf.
>>
>> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
>> > Thanks Harsh,
>> >
>> > For your comments, "What it means is that the NameNode need not store
>> > anything locally", you mean Primary Name Node do not need to store
>> > checkpoint/journal locally, and only need to keep memory image
>> > up-to-date
>> > for edits?
>> >
>> > regards,
>> > Lin
>> >
>> >
>> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>> >>
>> >> Hi Lin,
>> >>
>> >> My reply inline.
>> >>
>> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> >> > Hi guys,
>> >> >
>> >> > I am reading from this paper to learn about backup nodes
>> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >> >
>> >> > It is mentioned, "It contains all file system metadata information
>> >> > except
>> >> > for block locations. It can perform all operations of the regular
>> >> > NameNode
>> >> > that do not involve modification of the namespace or knowledge of
>> >> > block
>> >> > locations. ", what kinds of operations do not need knowledge of block
>> >> > locations?
>> >>
>> >> Operations that do not involve data reads or writes would not require
>> >> knowledge of block locations. Applying also the restriction of no
>> >> namespace mutation, an example would be listing directories and
>> >> looking up file information via FileStatus objects (perhaps the only
>> >> examples - its like a safemode but no reads either).
>> >>
>> >> > It is also mentioned, "Use of a BackupNode provides the option of
>> >> > running
>> >> > the NameNode without persistent storage, delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting to the BackupNode.", what means "running
>> >> > the
>> >> > NameNode without persistent storage" and "delegating responsibility
>> >> > for
>> >> > the
>> >> > namespace state persisting"?
>> >>
>> >> What it means is that the NameNode need not store anything locally,
>> >> but can rely on the edits being stored at the BackupNameNode which
>> >> would continuously be receiving it. When restarted, it can grab a
>> >> current checkpoint from the BNN and boot up anywhere, since there's no
>> >> local storage requirement.
>> >>
>> >> --
>> >> Harsh J
>> >
>> >
>>
>>
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

Hi Harsh,
Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?


On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks for answering my question, Harsh.

regards,
Lin

On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks for answering my question, Harsh.

regards,
Lin

On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

Hi Harsh,
Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?


On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

Hi Harsh,
Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?


On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Azuryy Yu <az...@gmail.com>]

Hi Harsh,
Do you mean BackupNameNode is Secondary NameNode in Hadoop1.x?


On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks for answering my question, Harsh.

regards,
Lin

On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks for answering my question, Harsh.

regards,
Lin

On Sun, Apr 7, 2013 at 4:05 PM, Harsh J <ha...@cloudera.com> wrote:

> Yes, it need not keep an edits (transactions) stream locally cause
> those are passed synchronously to the BackupNameNode, which persists
> it on its behalf.
>
> On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> > Thanks Harsh,
> >
> > For your comments, "What it means is that the NameNode need not store
> > anything locally", you mean Primary Name Node do not need to store
> > checkpoint/journal locally, and only need to keep memory image up-to-date
> > for edits?
> >
> > regards,
> > Lin
> >
> >
> > On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
> >>
> >> Hi Lin,
> >>
> >> My reply inline.
> >>
> >> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> >> > Hi guys,
> >> >
> >> > I am reading from this paper to learn about backup nodes
> >> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >> >
> >> > It is mentioned, "It contains all file system metadata information
> >> > except
> >> > for block locations. It can perform all operations of the regular
> >> > NameNode
> >> > that do not involve modification of the namespace or knowledge of
> block
> >> > locations. ", what kinds of operations do not need knowledge of block
> >> > locations?
> >>
> >> Operations that do not involve data reads or writes would not require
> >> knowledge of block locations. Applying also the restriction of no
> >> namespace mutation, an example would be listing directories and
> >> looking up file information via FileStatus objects (perhaps the only
> >> examples - its like a safemode but no reads either).
> >>
> >> > It is also mentioned, "Use of a BackupNode provides the option of
> >> > running
> >> > the NameNode without persistent storage, delegating responsibility for
> >> > the
> >> > namespace state persisting to the BackupNode.", what means "running
> the
> >> > NameNode without persistent storage" and "delegating responsibility
> for
> >> > the
> >> > namespace state persisting"?
> >>
> >> What it means is that the NameNode need not store anything locally,
> >> but can rely on the edits being stored at the BackupNameNode which
> >> would continuously be receiving it. When restarted, it can grab a
> >> current checkpoint from the BNN and boot up anywhere, since there's no
> >> local storage requirement.
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Yes, it need not keep an edits (transactions) stream locally cause
those are passed synchronously to the BackupNameNode, which persists
it on its behalf.

On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> Thanks Harsh,
>
> For your comments, "What it means is that the NameNode need not store
> anything locally", you mean Primary Name Node do not need to store
> checkpoint/journal locally, and only need to keep memory image up-to-date
> for edits?
>
> regards,
> Lin
>
>
> On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi Lin,
>>
>> My reply inline.
>>
>> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> > Hi guys,
>> >
>> > I am reading from this paper to learn about backup nodes
>> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >
>> > It is mentioned, "It contains all file system metadata information
>> > except
>> > for block locations. It can perform all operations of the regular
>> > NameNode
>> > that do not involve modification of the namespace or knowledge of block
>> > locations. ", what kinds of operations do not need knowledge of block
>> > locations?
>>
>> Operations that do not involve data reads or writes would not require
>> knowledge of block locations. Applying also the restriction of no
>> namespace mutation, an example would be listing directories and
>> looking up file information via FileStatus objects (perhaps the only
>> examples - its like a safemode but no reads either).
>>
>> > It is also mentioned, "Use of a BackupNode provides the option of
>> > running
>> > the NameNode without persistent storage, delegating responsibility for
>> > the
>> > namespace state persisting to the BackupNode.", what means "running the
>> > NameNode without persistent storage" and "delegating responsibility for
>> > the
>> > namespace state persisting"?
>>
>> What it means is that the NameNode need not store anything locally,
>> but can rely on the edits being stored at the BackupNameNode which
>> would continuously be receiving it. When restarted, it can grab a
>> current checkpoint from the BNN and boot up anywhere, since there's no
>> local storage requirement.
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Yes, it need not keep an edits (transactions) stream locally cause
those are passed synchronously to the BackupNameNode, which persists
it on its behalf.

On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> Thanks Harsh,
>
> For your comments, "What it means is that the NameNode need not store
> anything locally", you mean Primary Name Node do not need to store
> checkpoint/journal locally, and only need to keep memory image up-to-date
> for edits?
>
> regards,
> Lin
>
>
> On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi Lin,
>>
>> My reply inline.
>>
>> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> > Hi guys,
>> >
>> > I am reading from this paper to learn about backup nodes
>> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >
>> > It is mentioned, "It contains all file system metadata information
>> > except
>> > for block locations. It can perform all operations of the regular
>> > NameNode
>> > that do not involve modification of the namespace or knowledge of block
>> > locations. ", what kinds of operations do not need knowledge of block
>> > locations?
>>
>> Operations that do not involve data reads or writes would not require
>> knowledge of block locations. Applying also the restriction of no
>> namespace mutation, an example would be listing directories and
>> looking up file information via FileStatus objects (perhaps the only
>> examples - its like a safemode but no reads either).
>>
>> > It is also mentioned, "Use of a BackupNode provides the option of
>> > running
>> > the NameNode without persistent storage, delegating responsibility for
>> > the
>> > namespace state persisting to the BackupNode.", what means "running the
>> > NameNode without persistent storage" and "delegating responsibility for
>> > the
>> > namespace state persisting"?
>>
>> What it means is that the NameNode need not store anything locally,
>> but can rely on the edits being stored at the BackupNameNode which
>> would continuously be receiving it. When restarted, it can grab a
>> current checkpoint from the BNN and boot up anywhere, since there's no
>> local storage requirement.
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Yes, it need not keep an edits (transactions) stream locally cause
those are passed synchronously to the BackupNameNode, which persists
it on its behalf.

On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> Thanks Harsh,
>
> For your comments, "What it means is that the NameNode need not store
> anything locally", you mean Primary Name Node do not need to store
> checkpoint/journal locally, and only need to keep memory image up-to-date
> for edits?
>
> regards,
> Lin
>
>
> On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi Lin,
>>
>> My reply inline.
>>
>> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> > Hi guys,
>> >
>> > I am reading from this paper to learn about backup nodes
>> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >
>> > It is mentioned, "It contains all file system metadata information
>> > except
>> > for block locations. It can perform all operations of the regular
>> > NameNode
>> > that do not involve modification of the namespace or knowledge of block
>> > locations. ", what kinds of operations do not need knowledge of block
>> > locations?
>>
>> Operations that do not involve data reads or writes would not require
>> knowledge of block locations. Applying also the restriction of no
>> namespace mutation, an example would be listing directories and
>> looking up file information via FileStatus objects (perhaps the only
>> examples - its like a safemode but no reads either).
>>
>> > It is also mentioned, "Use of a BackupNode provides the option of
>> > running
>> > the NameNode without persistent storage, delegating responsibility for
>> > the
>> > namespace state persisting to the BackupNode.", what means "running the
>> > NameNode without persistent storage" and "delegating responsibility for
>> > the
>> > namespace state persisting"?
>>
>> What it means is that the NameNode need not store anything locally,
>> but can rely on the edits being stored at the BackupNameNode which
>> would continuously be receiving it. When restarted, it can grab a
>> current checkpoint from the BNN and boot up anywhere, since there's no
>> local storage requirement.
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Yes, it need not keep an edits (transactions) stream locally cause
those are passed synchronously to the BackupNameNode, which persists
it on its behalf.

On Sun, Apr 7, 2013 at 1:21 PM, Lin Ma <li...@gmail.com> wrote:
> Thanks Harsh,
>
> For your comments, "What it means is that the NameNode need not store
> anything locally", you mean Primary Name Node do not need to store
> checkpoint/journal locally, and only need to keep memory image up-to-date
> for edits?
>
> regards,
> Lin
>
>
> On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:
>>
>> Hi Lin,
>>
>> My reply inline.
>>
>> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
>> > Hi guys,
>> >
>> > I am reading from this paper to learn about backup nodes
>> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>> >
>> > It is mentioned, "It contains all file system metadata information
>> > except
>> > for block locations. It can perform all operations of the regular
>> > NameNode
>> > that do not involve modification of the namespace or knowledge of block
>> > locations. ", what kinds of operations do not need knowledge of block
>> > locations?
>>
>> Operations that do not involve data reads or writes would not require
>> knowledge of block locations. Applying also the restriction of no
>> namespace mutation, an example would be listing directories and
>> looking up file information via FileStatus objects (perhaps the only
>> examples - its like a safemode but no reads either).
>>
>> > It is also mentioned, "Use of a BackupNode provides the option of
>> > running
>> > the NameNode without persistent storage, delegating responsibility for
>> > the
>> > namespace state persisting to the BackupNode.", what means "running the
>> > NameNode without persistent storage" and "delegating responsibility for
>> > the
>> > namespace state persisting"?
>>
>> What it means is that the NameNode need not store anything locally,
>> but can rely on the edits being stored at the BackupNameNode which
>> would continuously be receiving it. When restarted, it can grab a
>> current checkpoint from the BNN and boot up anywhere, since there's no
>> local storage requirement.
>>
>> --
>> Harsh J
>
>



-- 
Harsh J

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks Harsh,

For your comments, "What it means is that the NameNode need not store
anything locally", you mean Primary Name Node do not need to store
checkpoint/journal locally, and only need to keep memory image up-to-date
for edits?

regards,
Lin

On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Lin,
>
> My reply inline.
>
> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> > Hi guys,
> >
> > I am reading from this paper to learn about backup nodes
> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >
> > It is mentioned, "It contains all file system metadata information except
> > for block locations. It can perform all operations of the regular
> NameNode
> > that do not involve modification of the namespace or knowledge of block
> > locations. ", what kinds of operations do not need knowledge of block
> > locations?
>
> Operations that do not involve data reads or writes would not require
> knowledge of block locations. Applying also the restriction of no
> namespace mutation, an example would be listing directories and
> looking up file information via FileStatus objects (perhaps the only
> examples - its like a safemode but no reads either).
>
> > It is also mentioned, "Use of a BackupNode provides the option of running
> > the NameNode without persistent storage, delegating responsibility for
> the
> > namespace state persisting to the BackupNode.", what means "running the
> > NameNode without persistent storage" and "delegating responsibility for
> the
> > namespace state persisting"?
>
> What it means is that the NameNode need not store anything locally,
> but can rely on the edits being stored at the BackupNameNode which
> would continuously be receiving it. When restarted, it can grab a
> current checkpoint from the BNN and boot up anywhere, since there's no
> local storage requirement.
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks Harsh,

For your comments, "What it means is that the NameNode need not store
anything locally", you mean Primary Name Node do not need to store
checkpoint/journal locally, and only need to keep memory image up-to-date
for edits?

regards,
Lin

On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Lin,
>
> My reply inline.
>
> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> > Hi guys,
> >
> > I am reading from this paper to learn about backup nodes
> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >
> > It is mentioned, "It contains all file system metadata information except
> > for block locations. It can perform all operations of the regular
> NameNode
> > that do not involve modification of the namespace or knowledge of block
> > locations. ", what kinds of operations do not need knowledge of block
> > locations?
>
> Operations that do not involve data reads or writes would not require
> knowledge of block locations. Applying also the restriction of no
> namespace mutation, an example would be listing directories and
> looking up file information via FileStatus objects (perhaps the only
> examples - its like a safemode but no reads either).
>
> > It is also mentioned, "Use of a BackupNode provides the option of running
> > the NameNode without persistent storage, delegating responsibility for
> the
> > namespace state persisting to the BackupNode.", what means "running the
> > NameNode without persistent storage" and "delegating responsibility for
> the
> > namespace state persisting"?
>
> What it means is that the NameNode need not store anything locally,
> but can rely on the edits being stored at the BackupNameNode which
> would continuously be receiving it. When restarted, it can grab a
> current checkpoint from the BNN and boot up anywhere, since there's no
> local storage requirement.
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks Harsh,

For your comments, "What it means is that the NameNode need not store
anything locally", you mean Primary Name Node do not need to store
checkpoint/journal locally, and only need to keep memory image up-to-date
for edits?

regards,
Lin

On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Lin,
>
> My reply inline.
>
> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> > Hi guys,
> >
> > I am reading from this paper to learn about backup nodes
> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >
> > It is mentioned, "It contains all file system metadata information except
> > for block locations. It can perform all operations of the regular
> NameNode
> > that do not involve modification of the namespace or knowledge of block
> > locations. ", what kinds of operations do not need knowledge of block
> > locations?
>
> Operations that do not involve data reads or writes would not require
> knowledge of block locations. Applying also the restriction of no
> namespace mutation, an example would be listing directories and
> looking up file information via FileStatus objects (perhaps the only
> examples - its like a safemode but no reads either).
>
> > It is also mentioned, "Use of a BackupNode provides the option of running
> > the NameNode without persistent storage, delegating responsibility for
> the
> > namespace state persisting to the BackupNode.", what means "running the
> > NameNode without persistent storage" and "delegating responsibility for
> the
> > namespace state persisting"?
>
> What it means is that the NameNode need not store anything locally,
> but can rely on the edits being stored at the BackupNameNode which
> would continuously be receiving it. When restarted, it can grab a
> current checkpoint from the BNN and boot up anywhere, since there's no
> local storage requirement.
>
> --
> Harsh J
>

Re: backup node question

[Lin Ma <li...@gmail.com>]

Thanks Harsh,

For your comments, "What it means is that the NameNode need not store
anything locally", you mean Primary Name Node do not need to store
checkpoint/journal locally, and only need to keep memory image up-to-date
for edits?

regards,
Lin

On Sun, Apr 7, 2013 at 3:31 PM, Harsh J <ha...@cloudera.com> wrote:

> Hi Lin,
>
> My reply inline.
>
> On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> > Hi guys,
> >
> > I am reading from this paper to learn about backup nodes
> > (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
> >
> > It is mentioned, "It contains all file system metadata information except
> > for block locations. It can perform all operations of the regular
> NameNode
> > that do not involve modification of the namespace or knowledge of block
> > locations. ", what kinds of operations do not need knowledge of block
> > locations?
>
> Operations that do not involve data reads or writes would not require
> knowledge of block locations. Applying also the restriction of no
> namespace mutation, an example would be listing directories and
> looking up file information via FileStatus objects (perhaps the only
> examples - its like a safemode but no reads either).
>
> > It is also mentioned, "Use of a BackupNode provides the option of running
> > the NameNode without persistent storage, delegating responsibility for
> the
> > namespace state persisting to the BackupNode.", what means "running the
> > NameNode without persistent storage" and "delegating responsibility for
> the
> > namespace state persisting"?
>
> What it means is that the NameNode need not store anything locally,
> but can rely on the edits being stored at the BackupNameNode which
> would continuously be receiving it. When restarted, it can grab a
> current checkpoint from the BNN and boot up anywhere, since there's no
> local storage requirement.
>
> --
> Harsh J
>

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Hi Lin,

My reply inline.

On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> Hi guys,
>
> I am reading from this paper to learn about backup nodes
> (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>
> It is mentioned, "It contains all file system metadata information except
> for block locations. It can perform all operations of the regular NameNode
> that do not involve modification of the namespace or knowledge of block
> locations. ", what kinds of operations do not need knowledge of block
> locations?

Operations that do not involve data reads or writes would not require
knowledge of block locations. Applying also the restriction of no
namespace mutation, an example would be listing directories and
looking up file information via FileStatus objects (perhaps the only
examples - its like a safemode but no reads either).

> It is also mentioned, "Use of a BackupNode provides the option of running
> the NameNode without persistent storage, delegating responsibility for the
> namespace state persisting to the BackupNode.", what means "running the
> NameNode without persistent storage" and "delegating responsibility for the
> namespace state persisting"?

What it means is that the NameNode need not store anything locally,
but can rely on the edits being stored at the BackupNameNode which
would continuously be receiving it. When restarted, it can grab a
current checkpoint from the BNN and boot up anywhere, since there's no
local storage requirement.

--
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Hi Lin,

My reply inline.

On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> Hi guys,
>
> I am reading from this paper to learn about backup nodes
> (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>
> It is mentioned, "It contains all file system metadata information except
> for block locations. It can perform all operations of the regular NameNode
> that do not involve modification of the namespace or knowledge of block
> locations. ", what kinds of operations do not need knowledge of block
> locations?

Operations that do not involve data reads or writes would not require
knowledge of block locations. Applying also the restriction of no
namespace mutation, an example would be listing directories and
looking up file information via FileStatus objects (perhaps the only
examples - its like a safemode but no reads either).

> It is also mentioned, "Use of a BackupNode provides the option of running
> the NameNode without persistent storage, delegating responsibility for the
> namespace state persisting to the BackupNode.", what means "running the
> NameNode without persistent storage" and "delegating responsibility for the
> namespace state persisting"?

What it means is that the NameNode need not store anything locally,
but can rely on the edits being stored at the BackupNameNode which
would continuously be receiving it. When restarted, it can grab a
current checkpoint from the BNN and boot up anywhere, since there's no
local storage requirement.

--
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Hi Lin,

My reply inline.

On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> Hi guys,
>
> I am reading from this paper to learn about backup nodes
> (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>
> It is mentioned, "It contains all file system metadata information except
> for block locations. It can perform all operations of the regular NameNode
> that do not involve modification of the namespace or knowledge of block
> locations. ", what kinds of operations do not need knowledge of block
> locations?

Operations that do not involve data reads or writes would not require
knowledge of block locations. Applying also the restriction of no
namespace mutation, an example would be listing directories and
looking up file information via FileStatus objects (perhaps the only
examples - its like a safemode but no reads either).

> It is also mentioned, "Use of a BackupNode provides the option of running
> the NameNode without persistent storage, delegating responsibility for the
> namespace state persisting to the BackupNode.", what means "running the
> NameNode without persistent storage" and "delegating responsibility for the
> namespace state persisting"?

What it means is that the NameNode need not store anything locally,
but can rely on the edits being stored at the BackupNameNode which
would continuously be receiving it. When restarted, it can grab a
current checkpoint from the BNN and boot up anywhere, since there's no
local storage requirement.

--
Harsh J

Re: backup node question

[Harsh J <ha...@cloudera.com>]

Hi Lin,

My reply inline.

On Sun, Apr 7, 2013 at 12:36 PM, Lin Ma <li...@gmail.com> wrote:
> Hi guys,
>
> I am reading from this paper to learn about backup nodes
> (http://www.storageconference.org/2010/Papers/MSST/Shvachko.pdf),
>
> It is mentioned, "It contains all file system metadata information except
> for block locations. It can perform all operations of the regular NameNode
> that do not involve modification of the namespace or knowledge of block
> locations. ", what kinds of operations do not need knowledge of block
> locations?

Operations that do not involve data reads or writes would not require
knowledge of block locations. Applying also the restriction of no
namespace mutation, an example would be listing directories and
looking up file information via FileStatus objects (perhaps the only
examples - its like a safemode but no reads either).

> It is also mentioned, "Use of a BackupNode provides the option of running
> the NameNode without persistent storage, delegating responsibility for the
> namespace state persisting to the BackupNode.", what means "running the
> NameNode without persistent storage" and "delegating responsibility for the
> namespace state persisting"?

What it means is that the NameNode need not store anything locally,
but can rely on the edits being stored at the BackupNameNode which
would continuously be receiving it. When restarted, it can grab a
current checkpoint from the BNN and boot up anywhere, since there's no
local storage requirement.

--
Harsh J