You are viewing a plain text version of this content. The canonical link for it is here.
Posted to hdfs-user@hadoop.apache.org by Vincent Boucher <vi...@gmail.com> on 2011/10/06 11:19:20 UTC

hdfs file - datanode association

Hello,

We are wondering if it is possible to require that the namenode redirects the 
blocks of the files of a given directory to a particular set of datanodes?

Our case is the following:

 - Servers
    10 x mass storage servers of each 50TB, RAID6
       -> 500TB available for hdfs
    30 x working nodes with 6TB (no RAID)
       -> 180TB available for hdfs

We'd like that the files stored in
   /hdfs/ms
be hosted on the mass storage (ms) servers,
while the files in
  /hdfs/wn
be hosted on the working nodes (wn).

Having 2 namenodes, one for each subdir /hdfs/ms or /hdfs/wn would do the job. 
However, at the client level it means dealing with to hadoop 
installation/libs ... not great.

It is possible to do it with a single namenode?

Cheers,

Vincent

Re: hdfs file - datanode association

Posted by Vincent Boucher <vi...@gmail.com>.
Hi John, Hi Will,

Thank you for your answers.

John,
>[...]
>If you have two different NNs, why would you need separate libs etc? You
>could point to the respective namenode while accessing that namenode.

From the point of view of the end-users, it should be transparent when they 
want to write from their code to the storage volumes to not bother with 
different configuration schemes.


Will,
>Why do you want to partition your datanodes this way? In our cluster, 
datanode size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an
>ideal world all datanodes would be similar in configuration, we have not
>observed any issues with this arrangement in production.

It's much more likely that a working node goes down or has its hosted data 
corrupted than a mass storage (with raid, redundant power sources & ethernet 
bonding). 
The data hosted on the working nodes, hadoop path: /hdfs/wn, have then to be 
replicated (replication factor = 2), while replication is not necessary for 
the data stored on the mass storage servers.




Le Thursday 06 October 2011, Will Maier a écrit :
> Hi Vincent-
>
> On Thu, Oct 06, 2011 at 11:19:20AM +0200, Vincent Boucher wrote:
> > We are wondering if it is possible to require that the namenode redirects
> > the blocks of the files of a given directory to a particular set of
> > datanodes?
> >
> > Our case is the following:
> >
> >  - Servers
> >     10 x mass storage servers of each 50TB, RAID6
> >        -> 500TB available for hdfs
> >     30 x working nodes with 6TB (no RAID)
> >        -> 180TB available for hdfs
> >
> > We'd like that the files stored in
> >    /hdfs/ms
> > be hosted on the mass storage (ms) servers,
> > while the files in
> >   /hdfs/wn
> > be hosted on the working nodes (wn).
>
> Why do you want to partition your datanodes this way? In our cluster,
> datanode size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an
> ideal world all datanodes would be similar in configuration, we have not
> observed any issues with this arrangement in production.


Re: hdfs file - datanode association

Posted by Vincent Boucher <vi...@gmail.com>.
It seems there's such a solution in 0.21, let's see how it integrates with 
0.20.203

https://issues.apache.org/jira/browse/HDFS-385
http://hadoopblog.blogspot.com/2009/09/hdfs-block-replica-placement-in-your.html

Do you have any hint about the release date of the stable 0.21th version?

Vincent

Le Thursday 06 October 2011, Will Maier a écrit :
> Hi Vincent-
>
> On Thu, Oct 06, 2011 at 11:19:20AM +0200, Vincent Boucher wrote:
> > We are wondering if it is possible to require that the namenode redirects
> > the blocks of the files of a given directory to a particular set of
> > datanodes?
> >
> > Our case is the following:
> >
> >  - Servers
> >     10 x mass storage servers of each 50TB, RAID6
> >        -> 500TB available for hdfs
> >     30 x working nodes with 6TB (no RAID)
> >        -> 180TB available for hdfs
> >
> > We'd like that the files stored in
> >    /hdfs/ms
> > be hosted on the mass storage (ms) servers,
> > while the files in
> >   /hdfs/wn
> > be hosted on the working nodes (wn).
>
> Why do you want to partition your datanodes this way? In our cluster,
> datanode size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an
> ideal world all datanodes would be similar in configuration, we have not
> observed any issues with this arrangement in production.


Re: hdfs file - datanode association

Posted by Will Maier <wc...@hep.wisc.edu>.
Hi Vincent-

On Thu, Oct 06, 2011 at 11:19:20AM +0200, Vincent Boucher wrote:
> We are wondering if it is possible to require that the namenode redirects the 
> blocks of the files of a given directory to a particular set of datanodes?
> 
> Our case is the following:
> 
>  - Servers
>     10 x mass storage servers of each 50TB, RAID6
>        -> 500TB available for hdfs
>     30 x working nodes with 6TB (no RAID)
>        -> 180TB available for hdfs
> 
> We'd like that the files stored in
>    /hdfs/ms
> be hosted on the mass storage (ms) servers,
> while the files in
>   /hdfs/wn
> be hosted on the working nodes (wn).

Why do you want to partition your datanodes this way? In our cluster, datanode
size ranges from ~1 TB (no RAID) up to ~80TB (RAID). While in an ideal world all
datanodes would be similar in configuration, we have not observed any issues
with this arrangement in production.

-- 

Will Maier - UW High Energy Physics
cel: 608.438.6162
tel: 608.263.9692
web: http://www.hep.wisc.edu/~wcmaier/

Re: hdfs file - datanode association

Posted by John George <jo...@yahoo-inc.com>.
AFAIK, it might not be possible with a single namenode. You could look
into federated namenodes - but I do not think even federated NN would
allow that today (I might be wrong though).

If you have two different NNs, why would you need separate libs etc? You
could point to the respective namenode while accessing that namenode.

-John George

On 10/6/11 4:19 AM, "Vincent Boucher" <vi...@gmail.com> wrote:

>Hello,
>
>We are wondering if it is possible to require that the namenode redirects
>the 
>blocks of the files of a given directory to a particular set of datanodes?
>
>Our case is the following:
>
> - Servers
>    10 x mass storage servers of each 50TB, RAID6
>       -> 500TB available for hdfs
>    30 x working nodes with 6TB (no RAID)
>       -> 180TB available for hdfs
>
>We'd like that the files stored in
>   /hdfs/ms
>be hosted on the mass storage (ms) servers,
>while the files in
>  /hdfs/wn
>be hosted on the working nodes (wn).
>
>Having 2 namenodes, one for each subdir /hdfs/ms or /hdfs/wn would do the
>job. 
>However, at the client level it means dealing with to hadoop
>installation/libs ... not great.
>
>It is possible to do it with a single namenode?
>
>Cheers,
>
>Vincent