You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Jeff Zhang <zj...@gmail.com> on 2010/06/11 09:18:36 UTC
How to make command setrepr take effect on data node ?
Hi all,
The command bin/hadoop setrepr only affect the metadata in namenode.
So How can I make it take effect on data node ? Thanks.
--
Best Regards
Jeff Zhang
Re: How to make command setrepr take effect on data node ?
Posted by Todd Lipcon <to...@cloudera.com>.
On Sat, Jun 12, 2010 at 8:18 PM, Jeff Zhang <zj...@gmail.com> wrote:
> So when will the replication happen ? Do you know which part of code
> control the replication ? Thanks.
>
>
It's all throughout FSNamesystem and related classes. I wouldn't recommend
digging in there unless you have a couple days to kill - it gets quite
hairy! :)
Replication should happen close to immediately unless there is already a
large queue of underreplicated blocks. Replication happens through a
priority queue, where the priority is based on how far under-replicated a
block is.
-Todd
>
> On Sun, Jun 13, 2010 at 11:10 AM, Eric Sammer <es...@cloudera.com>
> wrote:
> > There is no way to force replication other than setrep. Do you have
> > fewer nodes than configured replicas? Hadoop will not place two
> > replicas on one data node, for instance.
> >
> > On Sat, Jun 12, 2010 at 12:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
> >> But when I invoke command fsck, it show that one replica is missing.
> >> So how can I force the replication ?
> >>
> >> On Sat, Jun 12, 2010 at 12:13 PM, Eric Sammer <es...@cloudera.com>
> wrote:
> >>> Jeff:
> >>>
> >>> The name node will instruct the data nodes to do either remove or
> >>> create new replicas. There's nothing additional you need to do other
> >>> than fs -setrep.
> >>>
> >>> On Fri, Jun 11, 2010 at 3:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
> >>>> Hi all,
> >>>>
> >>>> The command bin/hadoop setrepr only affect the metadata in namenode.
> >>>> So How can I make it take effect on data node ? Thanks.
> >>>>
> >>>> --
> >>>> Best Regards
> >>>>
> >>>> Jeff Zhang
> >>>>
> >>>
> >>>
> >>>
> >>> --
> >>> Eric Sammer
> >>> phone: +1-917-287-2675
> >>> twitter: esammer
> >>> data: www.cloudera.com
> >>>
> >>
> >>
> >>
> >> --
> >> Best Regards
> >>
> >> Jeff Zhang
> >>
> >
> >
> >
> > --
> > Eric Sammer
> > phone: +1-917-287-2675
> > twitter: esammer
> > data: www.cloudera.com
> >
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: How to make command setrepr take effect on data node ?
Posted by Jeff Zhang <zj...@gmail.com>.
So when will the replication happen ? Do you know which part of code
control the replication ? Thanks.
On Sun, Jun 13, 2010 at 11:10 AM, Eric Sammer <es...@cloudera.com> wrote:
> There is no way to force replication other than setrep. Do you have
> fewer nodes than configured replicas? Hadoop will not place two
> replicas on one data node, for instance.
>
> On Sat, Jun 12, 2010 at 12:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
>> But when I invoke command fsck, it show that one replica is missing.
>> So how can I force the replication ?
>>
>> On Sat, Jun 12, 2010 at 12:13 PM, Eric Sammer <es...@cloudera.com> wrote:
>>> Jeff:
>>>
>>> The name node will instruct the data nodes to do either remove or
>>> create new replicas. There's nothing additional you need to do other
>>> than fs -setrep.
>>>
>>> On Fri, Jun 11, 2010 at 3:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
>>>> Hi all,
>>>>
>>>> The command bin/hadoop setrepr only affect the metadata in namenode.
>>>> So How can I make it take effect on data node ? Thanks.
>>>>
>>>> --
>>>> Best Regards
>>>>
>>>> Jeff Zhang
>>>>
>>>
>>>
>>>
>>> --
>>> Eric Sammer
>>> phone: +1-917-287-2675
>>> twitter: esammer
>>> data: www.cloudera.com
>>>
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>
--
Best Regards
Jeff Zhang
Re: How to make command setrepr take effect on data node ?
Posted by Eric Sammer <es...@cloudera.com>.
There is no way to force replication other than setrep. Do you have
fewer nodes than configured replicas? Hadoop will not place two
replicas on one data node, for instance.
On Sat, Jun 12, 2010 at 12:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
> But when I invoke command fsck, it show that one replica is missing.
> So how can I force the replication ?
>
> On Sat, Jun 12, 2010 at 12:13 PM, Eric Sammer <es...@cloudera.com> wrote:
>> Jeff:
>>
>> The name node will instruct the data nodes to do either remove or
>> create new replicas. There's nothing additional you need to do other
>> than fs -setrep.
>>
>> On Fri, Jun 11, 2010 at 3:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
>>> Hi all,
>>>
>>> The command bin/hadoop setrepr only affect the metadata in namenode.
>>> So How can I make it take effect on data node ? Thanks.
>>>
>>> --
>>> Best Regards
>>>
>>> Jeff Zhang
>>>
>>
>>
>>
>> --
>> Eric Sammer
>> phone: +1-917-287-2675
>> twitter: esammer
>> data: www.cloudera.com
>>
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com
Re: How to make command setrepr take effect on data node ?
Posted by Jeff Zhang <zj...@gmail.com>.
But when I invoke command fsck, it show that one replica is missing.
So how can I force the replication ?
On Sat, Jun 12, 2010 at 12:13 PM, Eric Sammer <es...@cloudera.com> wrote:
> Jeff:
>
> The name node will instruct the data nodes to do either remove or
> create new replicas. There's nothing additional you need to do other
> than fs -setrep.
>
> On Fri, Jun 11, 2010 at 3:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
>> Hi all,
>>
>> The command bin/hadoop setrepr only affect the metadata in namenode.
>> So How can I make it take effect on data node ? Thanks.
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>>
>
>
>
> --
> Eric Sammer
> phone: +1-917-287-2675
> twitter: esammer
> data: www.cloudera.com
>
--
Best Regards
Jeff Zhang
Re: How to make command setrepr take effect on data node ?
Posted by Eric Sammer <es...@cloudera.com>.
Jeff:
The name node will instruct the data nodes to do either remove or
create new replicas. There's nothing additional you need to do other
than fs -setrep.
On Fri, Jun 11, 2010 at 3:18 AM, Jeff Zhang <zj...@gmail.com> wrote:
> Hi all,
>
> The command bin/hadoop setrepr only affect the metadata in namenode.
> So How can I make it take effect on data node ? Thanks.
>
> --
> Best Regards
>
> Jeff Zhang
>
--
Eric Sammer
phone: +1-917-287-2675
twitter: esammer
data: www.cloudera.com