How to split the hdfs in different subgroups

[xa...@orange-ftgroup.com]

Hi There,

I have a hdfs and I want to split the cluster in two groups. Each groups
have a set of datanodes. I want to be able that my client (hdfshell)
only can write in one group. One group is in one rack and my other group
is in the other rack. Replication between racks is allowed but the
client has to read and write from one specific group. 
Is it possible?

I appreciate any help

Xavier

Re: How to split the hdfs in different subgroups

[Raghu Angadi <ra...@yahoo-inc.com>]

xavier.quintuna@orange-ftgroup.com wrote:
> I read the docs about rack awareness but my issue is how the client can
> pick some specific datanodes, which are located in some specific rack,
> to write the block there. The idea is that the client is able to write
> the block in two separated groups of datanodes in the same hdfs. For
> instance: bin/hadoop dfs -put <localpath> <dfspath> -location <rackid>

If you have just 2 racks, and replication of 2 or more, client will 
write at least one replica in each rack. And while reading, client will 
read the replica in located in its rack.

Currently you cannot explicitly control which rack reach replica goes.

Raghu.

RE: How to split the hdfs in different subgroups

[xa...@orange-ftgroup.com]

I read the docs about rack awareness but my issue is how the client can
pick some specific datanodes, which are located in some specific rack,
to write the block there. The idea is that the client is able to write
the block in two separated groups of datanodes in the same hdfs. For
instance: bin/hadoop dfs -put <localpath> <dfspath> -location <rackid>

Xavier



-----Original Message-----
From: Raghu Angadi [mailto:rangadi@yahoo-inc.com] 
Sent: Friday, February 22, 2008 10:59 AM
To: core-user@hadoop.apache.org
Subject: Re: How to split the hdfs in different subgroups


You could probably treat these two groups as different "racks". You can
read about rackawareness in
http://hadoop.apache.org/core/docs/r0.16.0/hdfs_user_guide.html , and
follow the links from there for more information regd how to configure
etc.

Raghu.

xavier.quintuna@orange-ftgroup.com wrote:
> Hi There,
> 
> I have a hdfs and I want to split the cluster in two groups. Each 
> groups have a set of datanodes. I want to be able that my client 
> (hdfshell) only can write in one group. One group is in one rack and 
> my other group is in the other rack. Replication between racks is 
> allowed but the client has to read and write from one specific group.
> Is it possible?
> 
> I appreciate any help
> 
> Xavier
> 

Re: How to split the hdfs in different subgroups

[Raghu Angadi <ra...@yahoo-inc.com>]

You could probably treat these two groups as different "racks". You can 
read about rackawareness in 
http://hadoop.apache.org/core/docs/r0.16.0/hdfs_user_guide.html , and 
follow the links from there for more information regd how to configure etc.

Raghu.

xavier.quintuna@orange-ftgroup.com wrote:
> Hi There,
> 
> I have a hdfs and I want to split the cluster in two groups. Each groups
> have a set of datanodes. I want to be able that my client (hdfshell)
> only can write in one group. One group is in one rack and my other group
> is in the other rack. Replication between racks is allowed but the
> client has to read and write from one specific group. 
> Is it possible?
> 
> I appreciate any help
> 
> Xavier
>