You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by oc tsdb <oc...@gmail.com> on 2013/12/09 12:52:44 UTC
Name node and data node replacement
Hi,
I am trying to understand,how to recover a Name node or Data node failure
in case if those nodes are unable to come up due to unexpected issues.
Is there a way to replace Name node with new name node without loosing
data.I mean new Name node should have all the data from original Name node.
Similarly for Data node.
Thanks
-OC
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Does NN log anywhere saying that 'one of DN is down and not reachable,so
replicating damaged DN data to another DN'?
Thanks
On Wed, Dec 11, 2013 at 2:10 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Yes you are right. It will periodically checks the under replicated blocks
> information and place those blocks on available data nodes if required.
>
> Regards
> Jitendra
>
>
> On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> You mean NN will always try to maintain number of replicas as configured
>> even if one DN goes down and unable to come up?
>>
>> Thanks
>> -OC
>>
>>
>>
>> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>>
>>> To recover a NN, you probably want to use the HA feature. If not, try
>>> writing your edits to a nfs volume in additional to the local fs.
>>>
>>> There is no need to recover a DN. The NN compensates for a lost DN by
>>> using the other remaining replicas to create a new replica on another DN.
>>>
>>> Daryn
>>>
>>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > I am trying to understand,how to recover a Name node or Data node
>>> failure in case if those nodes are unable to come up due to unexpected
>>> issues.
>>> >
>>> > Is there a way to replace Name node with new name node without loosing
>>> data.I mean new Name node should have all the data from original Name node.
>>> > Similarly for Data node.
>>> >
>>> >
>>> > Thanks
>>> > -OC
>>> >
>>> >
>>>
>>>
>>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Does NN log anywhere saying that 'one of DN is down and not reachable,so
replicating damaged DN data to another DN'?
Thanks
On Wed, Dec 11, 2013 at 2:10 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Yes you are right. It will periodically checks the under replicated blocks
> information and place those blocks on available data nodes if required.
>
> Regards
> Jitendra
>
>
> On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> You mean NN will always try to maintain number of replicas as configured
>> even if one DN goes down and unable to come up?
>>
>> Thanks
>> -OC
>>
>>
>>
>> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>>
>>> To recover a NN, you probably want to use the HA feature. If not, try
>>> writing your edits to a nfs volume in additional to the local fs.
>>>
>>> There is no need to recover a DN. The NN compensates for a lost DN by
>>> using the other remaining replicas to create a new replica on another DN.
>>>
>>> Daryn
>>>
>>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > I am trying to understand,how to recover a Name node or Data node
>>> failure in case if those nodes are unable to come up due to unexpected
>>> issues.
>>> >
>>> > Is there a way to replace Name node with new name node without loosing
>>> data.I mean new Name node should have all the data from original Name node.
>>> > Similarly for Data node.
>>> >
>>> >
>>> > Thanks
>>> > -OC
>>> >
>>> >
>>>
>>>
>>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Does NN log anywhere saying that 'one of DN is down and not reachable,so
replicating damaged DN data to another DN'?
Thanks
On Wed, Dec 11, 2013 at 2:10 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Yes you are right. It will periodically checks the under replicated blocks
> information and place those blocks on available data nodes if required.
>
> Regards
> Jitendra
>
>
> On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> You mean NN will always try to maintain number of replicas as configured
>> even if one DN goes down and unable to come up?
>>
>> Thanks
>> -OC
>>
>>
>>
>> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>>
>>> To recover a NN, you probably want to use the HA feature. If not, try
>>> writing your edits to a nfs volume in additional to the local fs.
>>>
>>> There is no need to recover a DN. The NN compensates for a lost DN by
>>> using the other remaining replicas to create a new replica on another DN.
>>>
>>> Daryn
>>>
>>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > I am trying to understand,how to recover a Name node or Data node
>>> failure in case if those nodes are unable to come up due to unexpected
>>> issues.
>>> >
>>> > Is there a way to replace Name node with new name node without loosing
>>> data.I mean new Name node should have all the data from original Name node.
>>> > Similarly for Data node.
>>> >
>>> >
>>> > Thanks
>>> > -OC
>>> >
>>> >
>>>
>>>
>>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Does NN log anywhere saying that 'one of DN is down and not reachable,so
replicating damaged DN data to another DN'?
Thanks
On Wed, Dec 11, 2013 at 2:10 PM, Jitendra Yadav
<je...@gmail.com>wrote:
> Yes you are right. It will periodically checks the under replicated blocks
> information and place those blocks on available data nodes if required.
>
> Regards
> Jitendra
>
>
> On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
>
>> Hi,
>>
>> Thanks for your response.
>>
>> You mean NN will always try to maintain number of replicas as configured
>> even if one DN goes down and unable to come up?
>>
>> Thanks
>> -OC
>>
>>
>>
>> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>>
>>> To recover a NN, you probably want to use the HA feature. If not, try
>>> writing your edits to a nfs volume in additional to the local fs.
>>>
>>> There is no need to recover a DN. The NN compensates for a lost DN by
>>> using the other remaining replicas to create a new replica on another DN.
>>>
>>> Daryn
>>>
>>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>>
>>> > Hi,
>>> >
>>> > I am trying to understand,how to recover a Name node or Data node
>>> failure in case if those nodes are unable to come up due to unexpected
>>> issues.
>>> >
>>> > Is there a way to replace Name node with new name node without loosing
>>> data.I mean new Name node should have all the data from original Name node.
>>> > Similarly for Data node.
>>> >
>>> >
>>> > Thanks
>>> > -OC
>>> >
>>> >
>>>
>>>
>>
>
Re: Name node and data node replacement
Posted by Jitendra Yadav <je...@gmail.com>.
Yes you are right. It will periodically checks the under replicated blocks
information and place those blocks on available data nodes if required.
Regards
Jitendra
On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> Thanks for your response.
>
> You mean NN will always try to maintain number of replicas as configured
> even if one DN goes down and unable to come up?
>
> Thanks
> -OC
>
>
>
> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>
>> To recover a NN, you probably want to use the HA feature. If not, try
>> writing your edits to a nfs volume in additional to the local fs.
>>
>> There is no need to recover a DN. The NN compensates for a lost DN by
>> using the other remaining replicas to create a new replica on another DN.
>>
>> Daryn
>>
>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand,how to recover a Name node or Data node
>> failure in case if those nodes are unable to come up due to unexpected
>> issues.
>> >
>> > Is there a way to replace Name node with new name node without loosing
>> data.I mean new Name node should have all the data from original Name node.
>> > Similarly for Data node.
>> >
>> >
>> > Thanks
>> > -OC
>> >
>> >
>>
>>
>
Re: Name node and data node replacement
Posted by Jitendra Yadav <je...@gmail.com>.
Yes you are right. It will periodically checks the under replicated blocks
information and place those blocks on available data nodes if required.
Regards
Jitendra
On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> Thanks for your response.
>
> You mean NN will always try to maintain number of replicas as configured
> even if one DN goes down and unable to come up?
>
> Thanks
> -OC
>
>
>
> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>
>> To recover a NN, you probably want to use the HA feature. If not, try
>> writing your edits to a nfs volume in additional to the local fs.
>>
>> There is no need to recover a DN. The NN compensates for a lost DN by
>> using the other remaining replicas to create a new replica on another DN.
>>
>> Daryn
>>
>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand,how to recover a Name node or Data node
>> failure in case if those nodes are unable to come up due to unexpected
>> issues.
>> >
>> > Is there a way to replace Name node with new name node without loosing
>> data.I mean new Name node should have all the data from original Name node.
>> > Similarly for Data node.
>> >
>> >
>> > Thanks
>> > -OC
>> >
>> >
>>
>>
>
Re: Name node and data node replacement
Posted by Jitendra Yadav <je...@gmail.com>.
Yes you are right. It will periodically checks the under replicated blocks
information and place those blocks on available data nodes if required.
Regards
Jitendra
On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> Thanks for your response.
>
> You mean NN will always try to maintain number of replicas as configured
> even if one DN goes down and unable to come up?
>
> Thanks
> -OC
>
>
>
> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>
>> To recover a NN, you probably want to use the HA feature. If not, try
>> writing your edits to a nfs volume in additional to the local fs.
>>
>> There is no need to recover a DN. The NN compensates for a lost DN by
>> using the other remaining replicas to create a new replica on another DN.
>>
>> Daryn
>>
>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand,how to recover a Name node or Data node
>> failure in case if those nodes are unable to come up due to unexpected
>> issues.
>> >
>> > Is there a way to replace Name node with new name node without loosing
>> data.I mean new Name node should have all the data from original Name node.
>> > Similarly for Data node.
>> >
>> >
>> > Thanks
>> > -OC
>> >
>> >
>>
>>
>
Re: Name node and data node replacement
Posted by Jitendra Yadav <je...@gmail.com>.
Yes you are right. It will periodically checks the under replicated blocks
information and place those blocks on available data nodes if required.
Regards
Jitendra
On Wed, Dec 11, 2013 at 1:57 PM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> Thanks for your response.
>
> You mean NN will always try to maintain number of replicas as configured
> even if one DN goes down and unable to come up?
>
> Thanks
> -OC
>
>
>
> On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
>
>> To recover a NN, you probably want to use the HA feature. If not, try
>> writing your edits to a nfs volume in additional to the local fs.
>>
>> There is no need to recover a DN. The NN compensates for a lost DN by
>> using the other remaining replicas to create a new replica on another DN.
>>
>> Daryn
>>
>> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>>
>> > Hi,
>> >
>> > I am trying to understand,how to recover a Name node or Data node
>> failure in case if those nodes are unable to come up due to unexpected
>> issues.
>> >
>> > Is there a way to replace Name node with new name node without loosing
>> data.I mean new Name node should have all the data from original Name node.
>> > Similarly for Data node.
>> >
>> >
>> > Thanks
>> > -OC
>> >
>> >
>>
>>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Thanks for your response.
You mean NN will always try to maintain number of replicas as configured
even if one DN goes down and unable to come up?
Thanks
-OC
On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> To recover a NN, you probably want to use the HA feature. If not, try
> writing your edits to a nfs volume in additional to the local fs.
>
> There is no need to recover a DN. The NN compensates for a lost DN by
> using the other remaining replicas to create a new replica on another DN.
>
> Daryn
>
> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to understand,how to recover a Name node or Data node
> failure in case if those nodes are unable to come up due to unexpected
> issues.
> >
> > Is there a way to replace Name node with new name node without loosing
> data.I mean new Name node should have all the data from original Name node.
> > Similarly for Data node.
> >
> >
> > Thanks
> > -OC
> >
> >
>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Thanks for your response.
You mean NN will always try to maintain number of replicas as configured
even if one DN goes down and unable to come up?
Thanks
-OC
On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> To recover a NN, you probably want to use the HA feature. If not, try
> writing your edits to a nfs volume in additional to the local fs.
>
> There is no need to recover a DN. The NN compensates for a lost DN by
> using the other remaining replicas to create a new replica on another DN.
>
> Daryn
>
> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to understand,how to recover a Name node or Data node
> failure in case if those nodes are unable to come up due to unexpected
> issues.
> >
> > Is there a way to replace Name node with new name node without loosing
> data.I mean new Name node should have all the data from original Name node.
> > Similarly for Data node.
> >
> >
> > Thanks
> > -OC
> >
> >
>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Thanks for your response.
You mean NN will always try to maintain number of replicas as configured
even if one DN goes down and unable to come up?
Thanks
-OC
On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> To recover a NN, you probably want to use the HA feature. If not, try
> writing your edits to a nfs volume in additional to the local fs.
>
> There is no need to recover a DN. The NN compensates for a lost DN by
> using the other remaining replicas to create a new replica on another DN.
>
> Daryn
>
> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to understand,how to recover a Name node or Data node
> failure in case if those nodes are unable to come up due to unexpected
> issues.
> >
> > Is there a way to replace Name node with new name node without loosing
> data.I mean new Name node should have all the data from original Name node.
> > Similarly for Data node.
> >
> >
> > Thanks
> > -OC
> >
> >
>
>
Re: Name node and data node replacement
Posted by oc tsdb <oc...@gmail.com>.
Hi,
Thanks for your response.
You mean NN will always try to maintain number of replicas as configured
even if one DN goes down and unable to come up?
Thanks
-OC
On Mon, Dec 9, 2013 at 8:18 PM, Daryn Sharp <da...@yahoo-inc.com> wrote:
> To recover a NN, you probably want to use the HA feature. If not, try
> writing your edits to a nfs volume in additional to the local fs.
>
> There is no need to recover a DN. The NN compensates for a lost DN by
> using the other remaining replicas to create a new replica on another DN.
>
> Daryn
>
> On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
>
> > Hi,
> >
> > I am trying to understand,how to recover a Name node or Data node
> failure in case if those nodes are unable to come up due to unexpected
> issues.
> >
> > Is there a way to replace Name node with new name node without loosing
> data.I mean new Name node should have all the data from original Name node.
> > Similarly for Data node.
> >
> >
> > Thanks
> > -OC
> >
> >
>
>
Re: Name node and data node replacement
Posted by Daryn Sharp <da...@yahoo-inc.com>.
To recover a NN, you probably want to use the HA feature. If not, try writing your edits to a nfs volume in additional to the local fs.
There is no need to recover a DN. The NN compensates for a lost DN by using the other remaining replicas to create a new replica on another DN.
Daryn
On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> I am trying to understand,how to recover a Name node or Data node failure in case if those nodes are unable to come up due to unexpected issues.
>
> Is there a way to replace Name node with new name node without loosing data.I mean new Name node should have all the data from original Name node.
> Similarly for Data node.
>
>
> Thanks
> -OC
>
>
Re: Name node and data node replacement
Posted by Daryn Sharp <da...@yahoo-inc.com>.
To recover a NN, you probably want to use the HA feature. If not, try writing your edits to a nfs volume in additional to the local fs.
There is no need to recover a DN. The NN compensates for a lost DN by using the other remaining replicas to create a new replica on another DN.
Daryn
On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> I am trying to understand,how to recover a Name node or Data node failure in case if those nodes are unable to come up due to unexpected issues.
>
> Is there a way to replace Name node with new name node without loosing data.I mean new Name node should have all the data from original Name node.
> Similarly for Data node.
>
>
> Thanks
> -OC
>
>
Re: Name node and data node replacement
Posted by Daryn Sharp <da...@yahoo-inc.com>.
To recover a NN, you probably want to use the HA feature. If not, try writing your edits to a nfs volume in additional to the local fs.
There is no need to recover a DN. The NN compensates for a lost DN by using the other remaining replicas to create a new replica on another DN.
Daryn
On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> I am trying to understand,how to recover a Name node or Data node failure in case if those nodes are unable to come up due to unexpected issues.
>
> Is there a way to replace Name node with new name node without loosing data.I mean new Name node should have all the data from original Name node.
> Similarly for Data node.
>
>
> Thanks
> -OC
>
>
Re: Name node and data node replacement
Posted by Daryn Sharp <da...@yahoo-inc.com>.
To recover a NN, you probably want to use the HA feature. If not, try writing your edits to a nfs volume in additional to the local fs.
There is no need to recover a DN. The NN compensates for a lost DN by using the other remaining replicas to create a new replica on another DN.
Daryn
On Dec 9, 2013, at 5:52 AM, oc tsdb <oc...@gmail.com> wrote:
> Hi,
>
> I am trying to understand,how to recover a Name node or Data node failure in case if those nodes are unable to come up due to unexpected issues.
>
> Is there a way to replace Name node with new name node without loosing data.I mean new Name node should have all the data from original Name node.
> Similarly for Data node.
>
>
> Thanks
> -OC
>
>