You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by "Ananth T. Sarathy" <an...@gmail.com> on 2010/01/28 03:28:51 UTC
Need to re replicate
One of our datanodes went bye bye. We added a bunch more data nodes, but
when I do a fsck i get a report that a bunch of files are only replicated on
2 server, which makes sense, because we had 3, and lost one. Now that we
have 6 more, is there anything i need to do replicate the those files are
will the cluster fix itself?
Ananth
Re: Need to re replicate
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Ananth -
Unfortunately, if your under-replication count isn't actively going down (at least by one per minute; on a large cluster, several hundred per minute), something is wrong.
Brian
On Jan 27, 2010, at 9:02 PM, Ananth T. Sarathy wrote:
> ok, it probably will take some time. I will check again in the morning!
> Thanks
>
> Ananth T Sarathy
>
>
> On Wed, Jan 27, 2010 at 10:00 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
>
>> Hey Ananth,
>>
>> Replication happens automatically. If it doesn't (should start within
>> seconds after the node is declared dead on the web interface), something is
>> wrong.
>>
>> Check your NN logfile for error messages.
>>
>> Brian
>>
>> On Jan 27, 2010, at 8:56 PM, Ananth T. Sarathy wrote:
>>
>>> when I run it, i get
>>>
>>> Time Stamp Iteration# Bytes Already Moved Bytes Left To
>>> Move Bytes Being Moved
>>> The cluster is balanced. Exiting...
>>>
>>> but the fsck is still giving me this
>>>
>>> /setup_procypherevaluation.exe: Under replicated
>>> blk_-6330660892317301772_3341. Target Replicas is 3 but found 2
>> replica(s).
>>>
>>> anyother ideas
>>>
>>> Ananth T Sarathy
>>>
>>>
>>> On Wed, Jan 27, 2010 at 9:45 PM, Raymond Jennings III <
>> raymondjiii@yahoo.com
>>>> wrote:
>>>
>>>> I would try running the rebalance utility. I would be curious to see
>> what
>>>> that will do and if that will fix it.
>>>>
>>>> --- On Wed, 1/27/10, Ananth T. Sarathy <an...@gmail.com>
>> wrote:
>>>>
>>>>> From: Ananth T. Sarathy <an...@gmail.com>
>>>>> Subject: Need to re replicate
>>>>> To: common-user@hadoop.apache.org
>>>>> Date: Wednesday, January 27, 2010, 9:28 PM
>>>>> One of our datanodes went bye bye. We
>>>>> added a bunch more data nodes, but
>>>>> when I do a fsck i get a report that a bunch of files are
>>>>> only replicated on
>>>>> 2 server, which makes sense, because we had 3, and lost
>>>>> one. Now that we
>>>>> have 6 more, is there anything i need to do replicate the
>>>>> those files are
>>>>> will the cluster fix itself?
>>>>> Ananth
>>>>>
>>>>
>>>>
>>>>
>>>>
>>
>>
Re: Need to re replicate
Posted by "Ananth T. Sarathy" <an...@gmail.com>.
ok, it probably will take some time. I will check again in the morning!
Thanks
Ananth T Sarathy
On Wed, Jan 27, 2010 at 10:00 PM, Brian Bockelman <bb...@cse.unl.edu>wrote:
> Hey Ananth,
>
> Replication happens automatically. If it doesn't (should start within
> seconds after the node is declared dead on the web interface), something is
> wrong.
>
> Check your NN logfile for error messages.
>
> Brian
>
> On Jan 27, 2010, at 8:56 PM, Ananth T. Sarathy wrote:
>
> > when I run it, i get
> >
> > Time Stamp Iteration# Bytes Already Moved Bytes Left To
> > Move Bytes Being Moved
> > The cluster is balanced. Exiting...
> >
> > but the fsck is still giving me this
> >
> > /setup_procypherevaluation.exe: Under replicated
> > blk_-6330660892317301772_3341. Target Replicas is 3 but found 2
> replica(s).
> >
> > anyother ideas
> >
> > Ananth T Sarathy
> >
> >
> > On Wed, Jan 27, 2010 at 9:45 PM, Raymond Jennings III <
> raymondjiii@yahoo.com
> >> wrote:
> >
> >> I would try running the rebalance utility. I would be curious to see
> what
> >> that will do and if that will fix it.
> >>
> >> --- On Wed, 1/27/10, Ananth T. Sarathy <an...@gmail.com>
> wrote:
> >>
> >>> From: Ananth T. Sarathy <an...@gmail.com>
> >>> Subject: Need to re replicate
> >>> To: common-user@hadoop.apache.org
> >>> Date: Wednesday, January 27, 2010, 9:28 PM
> >>> One of our datanodes went bye bye. We
> >>> added a bunch more data nodes, but
> >>> when I do a fsck i get a report that a bunch of files are
> >>> only replicated on
> >>> 2 server, which makes sense, because we had 3, and lost
> >>> one. Now that we
> >>> have 6 more, is there anything i need to do replicate the
> >>> those files are
> >>> will the cluster fix itself?
> >>> Ananth
> >>>
> >>
> >>
> >>
> >>
>
>
Re: Need to re replicate
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Ananth,
Replication happens automatically. If it doesn't (should start within seconds after the node is declared dead on the web interface), something is wrong.
Check your NN logfile for error messages.
Brian
On Jan 27, 2010, at 8:56 PM, Ananth T. Sarathy wrote:
> when I run it, i get
>
> Time Stamp Iteration# Bytes Already Moved Bytes Left To
> Move Bytes Being Moved
> The cluster is balanced. Exiting...
>
> but the fsck is still giving me this
>
> /setup_procypherevaluation.exe: Under replicated
> blk_-6330660892317301772_3341. Target Replicas is 3 but found 2 replica(s).
>
> anyother ideas
>
> Ananth T Sarathy
>
>
> On Wed, Jan 27, 2010 at 9:45 PM, Raymond Jennings III <raymondjiii@yahoo.com
>> wrote:
>
>> I would try running the rebalance utility. I would be curious to see what
>> that will do and if that will fix it.
>>
>> --- On Wed, 1/27/10, Ananth T. Sarathy <an...@gmail.com> wrote:
>>
>>> From: Ananth T. Sarathy <an...@gmail.com>
>>> Subject: Need to re replicate
>>> To: common-user@hadoop.apache.org
>>> Date: Wednesday, January 27, 2010, 9:28 PM
>>> One of our datanodes went bye bye. We
>>> added a bunch more data nodes, but
>>> when I do a fsck i get a report that a bunch of files are
>>> only replicated on
>>> 2 server, which makes sense, because we had 3, and lost
>>> one. Now that we
>>> have 6 more, is there anything i need to do replicate the
>>> those files are
>>> will the cluster fix itself?
>>> Ananth
>>>
>>
>>
>>
>>
Re: Need to re replicate
Posted by "Ananth T. Sarathy" <an...@gmail.com>.
when I run it, i get
Time Stamp Iteration# Bytes Already Moved Bytes Left To
Move Bytes Being Moved
The cluster is balanced. Exiting...
but the fsck is still giving me this
/setup_procypherevaluation.exe: Under replicated
blk_-6330660892317301772_3341. Target Replicas is 3 but found 2 replica(s).
anyother ideas
Ananth T Sarathy
On Wed, Jan 27, 2010 at 9:45 PM, Raymond Jennings III <raymondjiii@yahoo.com
> wrote:
> I would try running the rebalance utility. I would be curious to see what
> that will do and if that will fix it.
>
> --- On Wed, 1/27/10, Ananth T. Sarathy <an...@gmail.com> wrote:
>
> > From: Ananth T. Sarathy <an...@gmail.com>
> > Subject: Need to re replicate
> > To: common-user@hadoop.apache.org
> > Date: Wednesday, January 27, 2010, 9:28 PM
> > One of our datanodes went bye bye. We
> > added a bunch more data nodes, but
> > when I do a fsck i get a report that a bunch of files are
> > only replicated on
> > 2 server, which makes sense, because we had 3, and lost
> > one. Now that we
> > have 6 more, is there anything i need to do replicate the
> > those files are
> > will the cluster fix itself?
> > Ananth
> >
>
>
>
>
Re: Need to re replicate
Posted by Raymond Jennings III <ra...@yahoo.com>.
I would try running the rebalance utility. I would be curious to see what that will do and if that will fix it.
--- On Wed, 1/27/10, Ananth T. Sarathy <an...@gmail.com> wrote:
> From: Ananth T. Sarathy <an...@gmail.com>
> Subject: Need to re replicate
> To: common-user@hadoop.apache.org
> Date: Wednesday, January 27, 2010, 9:28 PM
> One of our datanodes went bye bye. We
> added a bunch more data nodes, but
> when I do a fsck i get a report that a bunch of files are
> only replicated on
> 2 server, which makes sense, because we had 3, and lost
> one. Now that we
> have 6 more, is there anything i need to do replicate the
> those files are
> will the cluster fix itself?
> Ananth
>