You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Giulio D'Ippolito <gi...@gmail.com> on 2012/07/04 12:37:59 UTC
Block location in HDFS
Hello,
I would like to insert in the filesystem some blocks, but each of these
blocks must be in a different storage point is there a way to specify this?
Thanks
Re: Block location in HDFS
Posted by Giulio D'Ippolito <gi...@gmail.com>.
Thanks, you get my problem in your second answer, the placement could be
randomly but must be unique for each block of the subset. I willl
investigate in more details your suggestions. For the privacy of my team
work i will not share this information in a mailing-list, If you like I
willl send you a PM.
Thanks.
Giulio
2012/7/4 Harsh J <ha...@cloudera.com>
> Right, but do you have a specific 4 nodes in mind when you do the
> write? If not, you may as well write to HDFS from a non-DN node and
> the block locations will be chosen randomly (with the default policy).
> This doesn't _guarantee_ that you get different locations for each
> block allocation, but does make the placement random.
>
> Otherwise, I suppose HDFS-385 is what you need, and your placement
> class can probably just do a round robin assignment of blocks every
> time.
>
> Just curious - why do you seek this?
>
> On Wed, Jul 4, 2012 at 8:31 PM, Giulio D'Ippolito
> <gi...@gmail.com> wrote:
> > Thanks for the answer,
> >
> > My problem is slightly different, let say that I have 6 storage point
> and i
> > have 4 files/blocks to insert in the filesystem, i would like that these
> > blocks will be placed in 4 different storage points, and of course i
> would
> > like to do the same for all the next 4 blocks. I'm not interested to the
> > replicas for the moment.
> > Thanks
> >
> >
> > Giulio
> >
> >
> >
> > 2012/7/4 Harsh J <ha...@cloudera.com>
> >>
> >> With the default replication/block placement policy, all replicas are
> >> anyway kept in separate nodes. Is that what you want?
> >>
> >> Or are you looking to specify a list of locations you want your blocks
> >> to go to? For the latter, there's no easy way but you may be
> >> interested in https://issues.apache.org/jira/browse/HDFS-385. But you
> >> can't send meta-info with each write with hints even with HDFS-385,
> >> and may need an external system.
> >>
> >> On Wed, Jul 4, 2012 at 4:07 PM, Giulio D'Ippolito
> >> <gi...@gmail.com> wrote:
> >> > Hello,
> >> >
> >> > I would like to insert in the filesystem some blocks, but each of
> these
> >> > blocks must be in a different storage point is there a way to specify
> >> > this?
> >> >
> >> > Thanks
> >>
> >>
> >>
> >> --
> >> Harsh J
> >
> >
>
>
>
> --
> Harsh J
>
Re: Block location in HDFS
Posted by Harsh J <ha...@cloudera.com>.
Right, but do you have a specific 4 nodes in mind when you do the
write? If not, you may as well write to HDFS from a non-DN node and
the block locations will be chosen randomly (with the default policy).
This doesn't _guarantee_ that you get different locations for each
block allocation, but does make the placement random.
Otherwise, I suppose HDFS-385 is what you need, and your placement
class can probably just do a round robin assignment of blocks every
time.
Just curious - why do you seek this?
On Wed, Jul 4, 2012 at 8:31 PM, Giulio D'Ippolito
<gi...@gmail.com> wrote:
> Thanks for the answer,
>
> My problem is slightly different, let say that I have 6 storage point and i
> have 4 files/blocks to insert in the filesystem, i would like that these
> blocks will be placed in 4 different storage points, and of course i would
> like to do the same for all the next 4 blocks. I'm not interested to the
> replicas for the moment.
> Thanks
>
>
> Giulio
>
>
>
> 2012/7/4 Harsh J <ha...@cloudera.com>
>>
>> With the default replication/block placement policy, all replicas are
>> anyway kept in separate nodes. Is that what you want?
>>
>> Or are you looking to specify a list of locations you want your blocks
>> to go to? For the latter, there's no easy way but you may be
>> interested in https://issues.apache.org/jira/browse/HDFS-385. But you
>> can't send meta-info with each write with hints even with HDFS-385,
>> and may need an external system.
>>
>> On Wed, Jul 4, 2012 at 4:07 PM, Giulio D'Ippolito
>> <gi...@gmail.com> wrote:
>> > Hello,
>> >
>> > I would like to insert in the filesystem some blocks, but each of these
>> > blocks must be in a different storage point is there a way to specify
>> > this?
>> >
>> > Thanks
>>
>>
>>
>> --
>> Harsh J
>
>
--
Harsh J
Re: Block location in HDFS
Posted by Giulio D'Ippolito <gi...@gmail.com>.
Thanks for the answer,
My problem is slightly different, let say that I have 6 storage point and i
have 4 files/blocks to insert in the filesystem, i would like that these
blocks will be placed in 4 different storage points, and of course i would
like to do the same for all the next 4 blocks. I'm not interested to the
replicas for the moment.
Thanks
Giulio
2012/7/4 Harsh J <ha...@cloudera.com>
> With the default replication/block placement policy, all replicas are
> anyway kept in separate nodes. Is that what you want?
>
> Or are you looking to specify a list of locations you want your blocks
> to go to? For the latter, there's no easy way but you may be
> interested in https://issues.apache.org/jira/browse/HDFS-385. But you
> can't send meta-info with each write with hints even with HDFS-385,
> and may need an external system.
>
> On Wed, Jul 4, 2012 at 4:07 PM, Giulio D'Ippolito
> <gi...@gmail.com> wrote:
> > Hello,
> >
> > I would like to insert in the filesystem some blocks, but each of these
> > blocks must be in a different storage point is there a way to specify
> this?
> >
> > Thanks
>
>
>
> --
> Harsh J
>
Re: Block location in HDFS
Posted by Harsh J <ha...@cloudera.com>.
With the default replication/block placement policy, all replicas are
anyway kept in separate nodes. Is that what you want?
Or are you looking to specify a list of locations you want your blocks
to go to? For the latter, there's no easy way but you may be
interested in https://issues.apache.org/jira/browse/HDFS-385. But you
can't send meta-info with each write with hints even with HDFS-385,
and may need an external system.
On Wed, Jul 4, 2012 at 4:07 PM, Giulio D'Ippolito
<gi...@gmail.com> wrote:
> Hello,
>
> I would like to insert in the filesystem some blocks, but each of these
> blocks must be in a different storage point is there a way to specify this?
>
> Thanks
--
Harsh J