You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-dev@hadoop.apache.org by Lipeng Wan <li...@gmail.com> on 2015/03/03 20:23:44 UTC
Dose hdfs support the configuration that different blocks can have
different number of replcias?
Hi devs,
By default, hdfs creates same number of replicas for each block. Is it
possible for us to create more replicas for some of the blocks?
Thanks!
L. W.
Re: Dose hdfs support the configuration that different blocks can
have different number of replcias?
Posted by Andrew Wang <an...@cloudera.com>.
Lipeng,
-setrep allows you to change the replication of an existing file. You can
also specify the replication factor when you initially create a file. Not
sure what you mean by "dynamically", that to me means calling setrep.
There is replication or invalidation work done as part of running -setrep.
This is done as a low-priority operation, unless the file is already in a
bad replication state (e.g. under-replicated).
Best,
Andrew
On Wed, Mar 4, 2015 at 12:18 PM, Lipeng Wan <li...@gmail.com> wrote:
> Hi Andrew,
>
> By using the -setrep command, can we change the replication factor of
> existing files? Or, can we change the replication factor of files
> dynamically? If that's possible, how much data movement overhead will
> occur?
> Thanks!
>
> Lipeng
>
> On Tue, Mar 3, 2015 at 2:57 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Yup, definitely. Check out the -setrep command:
> >
> >
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep
> >
> > HTH,
> > Andrew
> >
> > On Tue, Mar 3, 2015 at 11:49 AM, Lipeng Wan <li...@gmail.com>
> wrote:
> >
> >> Hi Andrew,
> >>
> >> Thanks for your reply!
> >> Then is it possible for us to specify different replication factors
> >> for different files?
> >>
> >> Lipeng
> >>
> >> On Tue, Mar 3, 2015 at 2:38 PM, Andrew Wang <an...@cloudera.com>
> >> wrote:
> >> > Hi Lipeng,
> >> >
> >> > Right now that is unsupported, replication is set on a per-file basis,
> >> not
> >> > per-block.
> >> >
> >> > Andrew
> >> >
> >> > On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <li...@gmail.com>
> >> wrote:
> >> >
> >> >> Hi devs,
> >> >>
> >> >> By default, hdfs creates same number of replicas for each block. Is
> it
> >> >> possible for us to create more replicas for some of the blocks?
> >> >> Thanks!
> >> >>
> >> >> L. W.
> >> >>
> >>
>
Re: Dose hdfs support the configuration that different blocks can
have different number of replcias?
Posted by Lipeng Wan <li...@gmail.com>.
Hi Andrew,
By using the -setrep command, can we change the replication factor of
existing files? Or, can we change the replication factor of files
dynamically? If that's possible, how much data movement overhead will
occur?
Thanks!
Lipeng
On Tue, Mar 3, 2015 at 2:57 PM, Andrew Wang <an...@cloudera.com> wrote:
> Yup, definitely. Check out the -setrep command:
>
> http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep
>
> HTH,
> Andrew
>
> On Tue, Mar 3, 2015 at 11:49 AM, Lipeng Wan <li...@gmail.com> wrote:
>
>> Hi Andrew,
>>
>> Thanks for your reply!
>> Then is it possible for us to specify different replication factors
>> for different files?
>>
>> Lipeng
>>
>> On Tue, Mar 3, 2015 at 2:38 PM, Andrew Wang <an...@cloudera.com>
>> wrote:
>> > Hi Lipeng,
>> >
>> > Right now that is unsupported, replication is set on a per-file basis,
>> not
>> > per-block.
>> >
>> > Andrew
>> >
>> > On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <li...@gmail.com>
>> wrote:
>> >
>> >> Hi devs,
>> >>
>> >> By default, hdfs creates same number of replicas for each block. Is it
>> >> possible for us to create more replicas for some of the blocks?
>> >> Thanks!
>> >>
>> >> L. W.
>> >>
>>
Re: Dose hdfs support the configuration that different blocks can
have different number of replcias?
Posted by Andrew Wang <an...@cloudera.com>.
Yup, definitely. Check out the -setrep command:
http://hadoop.apache.org/docs/current/hadoop-project-dist/hadoop-common/FileSystemShell.html#setrep
HTH,
Andrew
On Tue, Mar 3, 2015 at 11:49 AM, Lipeng Wan <li...@gmail.com> wrote:
> Hi Andrew,
>
> Thanks for your reply!
> Then is it possible for us to specify different replication factors
> for different files?
>
> Lipeng
>
> On Tue, Mar 3, 2015 at 2:38 PM, Andrew Wang <an...@cloudera.com>
> wrote:
> > Hi Lipeng,
> >
> > Right now that is unsupported, replication is set on a per-file basis,
> not
> > per-block.
> >
> > Andrew
> >
> > On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <li...@gmail.com>
> wrote:
> >
> >> Hi devs,
> >>
> >> By default, hdfs creates same number of replicas for each block. Is it
> >> possible for us to create more replicas for some of the blocks?
> >> Thanks!
> >>
> >> L. W.
> >>
>
Re: Dose hdfs support the configuration that different blocks can
have different number of replcias?
Posted by Lipeng Wan <li...@gmail.com>.
Hi Andrew,
Thanks for your reply!
Then is it possible for us to specify different replication factors
for different files?
Lipeng
On Tue, Mar 3, 2015 at 2:38 PM, Andrew Wang <an...@cloudera.com> wrote:
> Hi Lipeng,
>
> Right now that is unsupported, replication is set on a per-file basis, not
> per-block.
>
> Andrew
>
> On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <li...@gmail.com> wrote:
>
>> Hi devs,
>>
>> By default, hdfs creates same number of replicas for each block. Is it
>> possible for us to create more replicas for some of the blocks?
>> Thanks!
>>
>> L. W.
>>
Re: Dose hdfs support the configuration that different blocks can
have different number of replcias?
Posted by Andrew Wang <an...@cloudera.com>.
Hi Lipeng,
Right now that is unsupported, replication is set on a per-file basis, not
per-block.
Andrew
On Tue, Mar 3, 2015 at 11:23 AM, Lipeng Wan <li...@gmail.com> wrote:
> Hi devs,
>
> By default, hdfs creates same number of replicas for each block. Is it
> possible for us to create more replicas for some of the blocks?
> Thanks!
>
> L. W.
>