You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by C G <pa...@yahoo.com> on 2007/09/13 15:00:44 UTC
Multiple HDFS paths and Multiple Masters...
Hi All:
Two quick questions, thanks for any guidance...
I'd like to run nodes with around 2T of local disk set up as JBOD. So I would have 4 separate file systems per machine, for example /hdfs_a, /hdfs_b, /hdfs_c, /hdfs_d . Is it possible to configure things so that HDFS knows about all 4 file systems? Since we're using HDFS replication I see no point in using RAID-anything...to me that's the whole point of replication Comments?
Is it possible to set things up in Hadoop to run multiple masters? Is there any point/benefit reason for so-doing? Can I run multiple namenodes to guard against a single namenode going down or being wrecked?
If you can't run multiple namenodes, then that sort of implies the machine which is hosting *the* namenode needs to do all the traditional things to protect against data loss/corruption, including frequent backups, RAID mirroring, etc.
Thanks,
C G
---------------------------------
Looking for a deal? Find great prices on flights and hotels with Yahoo! FareChase.
Re: Multiple HDFS paths and Multiple Masters...
Posted by Hadoop <sa...@pearsonwholesale.com>.
Not sure if this will help but in my install 0.14.1 there is an option in
the conf/hadoop-default.xml
<property>
<name>dfs.data.dir</name>
<value>${hadoop.tmp.dir}/dfs/data</value>
<description>Determines where on the local filesystem an DFS data node
should store its blocks. If this is a comma-delimited
list of directories, then data will be stored in all named
directories, typically on different devices.
Directories that do not exist are ignored.
</description>
</property>
this suggest you shouldb e abile to list each of the paths there. hope this
helps.
"C G" <pa...@yahoo.com> wrote in message
news:322565.45799.qm@web45406.mail.sp1.yahoo.com...
> Hi All:
>
> Two quick questions, thanks for any guidance...
>
> I'd like to run nodes with around 2T of local disk set up as JBOD. So I
> would have 4 separate file systems per machine, for example /hdfs_a,
> /hdfs_b, /hdfs_c, /hdfs_d . Is it possible to configure things so that
> HDFS knows about all 4 file systems? Since we're using HDFS replication I
> see no point in using RAID-anything...to me that's the whole point of
> replication Comments?
>
> Is it possible to set things up in Hadoop to run multiple masters? Is
> there any point/benefit reason for so-doing? Can I run multiple namenodes
> to guard against a single namenode going down or being wrecked?
>
> If you can't run multiple namenodes, then that sort of implies the
> machine which is hosting *the* namenode needs to do all the traditional
> things to protect against data loss/corruption, including frequent
> backups, RAID mirroring, etc.
>
> Thanks,
> C G
>
>
>
> ---------------------------------
> Looking for a deal? Find great prices on flights and hotels with Yahoo!
> FareChase.
RE: Multiple HDFS paths and Multiple Masters...
Posted by Joydeep Sen Sarma <js...@facebook.com>.
I used to work in Netapp HA group - so can explain the single drive
failure stuff a bit (although the right forum is the
toasters@mathworks.com mailing list).
The shelves are supposed to bypass failed drives (the shelf re-routes
the FC loop when it detects failed drives). However, there were rare
drive failure modes where the drive would malfunction - but not in a way
detectable by the shelf - leading to the entire FC-loop malfunctioning -
and leading to multi-disk failure. The disks are dual attached - but in
this failure mode - they would take out both loops.
That said - this is circa 2004/5. New generation shelves fixed the
problems, as well as Netapp was asking shelf vendors for software
interface to power-cycle failed drives (so that netapp software could
take out bad drives by hard power-reset instead of relying on shelf
firmware). I don't know the current status.
In general - one of the big value adds of using Netapp (or EMC for that
matter) is their extensive understanding of drive/shelf failure modes
and the ability to proactively predict and take safeguard actions
against such failures.
Regarding RAID mirroring - well - it actually protects against cases
like this (since Netapp always puts mirrored copies on different
shelves/loops - thereby protecting against shelf/loop failure). But
RAID-4/5 (or netapp dual-parity) with backups and/or replication is a
good alternative (with somewhat lower availability guarantees and
performance)
Hope this helps ..
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thursday, September 13, 2007 9:47 AM
To: hadoop-user@lucene.apache.org
Subject: Re: Multiple HDFS paths and Multiple Masters...
Allen, Ted:
Good stuff...thanks for the information. Ted, a bit off-topic but
your comment about netapp single drive failures gave me pause,
particularly since we have a large one deployed now. Would you mind
saying more on that...feel free to contact me direct since it is
off-topic.
Thanks!
C G
Ted Dunning <td...@veoh.com> wrote:
On 9/13/07 6:00 AM, "C G"
wrote:
> I'd like to run nodes with around 2T of local disk set up as JBOD. So
I
> would have 4 separate file systems per machine, for example /hdfs_a,
/hdfs_b,
> /hdfs_c, /hdfs_d . Is it possible to configure things so that HDFS
knows
> about all 4 file systems?
Yes. This is normally done to allow heterogeneity in data/task nodes.
You
make a list of all of the file systems that MIGHT be available and
hadoop
figures out which are available and which have space to use.
> Since we're using HDFS replication I see no point in
> using RAID-anything...to me that's the whole point of replication
Comments?
That is the intent!
> Is it possible to set things up in Hadoop to run multiple masters?
Not yet. Doug makes very good points on this topic that a single master
will be fairly reliable and that it is the cluster that will have common
failures and thus must be robust to node failure.
There are lots of HA options. One that looks very nice to me (but that I
haven't tried) is DRDB which is a block level disk replication service.
See
http://www.drbd.org/ for more information (and let us know how it
looks).
The secondary nameserver may be of some help in recovery as well, but it
is
unlikely to be as quick as a replicated disk and a CARP based IP
address.
> If you can't run multiple namenodes, then that sort of implies the
machine
> which is hosting *the* namenode needs to do all the traditional things
to
> protect against data loss/corruption, including frequent backups, RAID
> mirroring, etc.
Some of these things are happening already, but the others are not a bad
idea at all. Consider your hardware carefully. RAID mirroring can
*decrease* reliability if you get a failure from either drive. Happened
to
me on my home machine and I have heard of other cases as well. Even in
sophisticated implementations such as are done by Netapp, you can have
drive
failures that freeze an entire shelf. My preference any more is
replicated
simple machines rather than fancy machines.
---------------------------------
Don't let your dream ride pass you by. Make it a reality with Yahoo!
Autos.
Re: Multiple HDFS paths and Multiple Masters...
Posted by C G <pa...@yahoo.com>.
Allen, Ted:
Good stuff...thanks for the information. Ted, a bit off-topic but your comment about netapp single drive failures gave me pause, particularly since we have a large one deployed now. Would you mind saying more on that...feel free to contact me direct since it is off-topic.
Thanks!
C G
Ted Dunning <td...@veoh.com> wrote:
On 9/13/07 6:00 AM, "C G"
wrote:
> I'd like to run nodes with around 2T of local disk set up as JBOD. So I
> would have 4 separate file systems per machine, for example /hdfs_a, /hdfs_b,
> /hdfs_c, /hdfs_d . Is it possible to configure things so that HDFS knows
> about all 4 file systems?
Yes. This is normally done to allow heterogeneity in data/task nodes. You
make a list of all of the file systems that MIGHT be available and hadoop
figures out which are available and which have space to use.
> Since we're using HDFS replication I see no point in
> using RAID-anything...to me that's the whole point of replication Comments?
That is the intent!
> Is it possible to set things up in Hadoop to run multiple masters?
Not yet. Doug makes very good points on this topic that a single master
will be fairly reliable and that it is the cluster that will have common
failures and thus must be robust to node failure.
There are lots of HA options. One that looks very nice to me (but that I
haven't tried) is DRDB which is a block level disk replication service. See
http://www.drbd.org/ for more information (and let us know how it looks).
The secondary nameserver may be of some help in recovery as well, but it is
unlikely to be as quick as a replicated disk and a CARP based IP address.
> If you can't run multiple namenodes, then that sort of implies the machine
> which is hosting *the* namenode needs to do all the traditional things to
> protect against data loss/corruption, including frequent backups, RAID
> mirroring, etc.
Some of these things are happening already, but the others are not a bad
idea at all. Consider your hardware carefully. RAID mirroring can
*decrease* reliability if you get a failure from either drive. Happened to
me on my home machine and I have heard of other cases as well. Even in
sophisticated implementations such as are done by Netapp, you can have drive
failures that freeze an entire shelf. My preference any more is replicated
simple machines rather than fancy machines.
---------------------------------
Don't let your dream ride pass you by. Make it a reality with Yahoo! Autos.
Re: Multiple HDFS paths and Multiple Masters...
Posted by Ted Dunning <td...@veoh.com>.
On 9/13/07 6:00 AM, "C G" <pa...@yahoo.com> wrote:
> I'd like to run nodes with around 2T of local disk set up as JBOD. So I
> would have 4 separate file systems per machine, for example /hdfs_a, /hdfs_b,
> /hdfs_c, /hdfs_d . Is it possible to configure things so that HDFS knows
> about all 4 file systems?
Yes. This is normally done to allow heterogeneity in data/task nodes. You
make a list of all of the file systems that MIGHT be available and hadoop
figures out which are available and which have space to use.
> Since we're using HDFS replication I see no point in
> using RAID-anything...to me that's the whole point of replication Comments?
That is the intent!
> Is it possible to set things up in Hadoop to run multiple masters?
Not yet. Doug makes very good points on this topic that a single master
will be fairly reliable and that it is the cluster that will have common
failures and thus must be robust to node failure.
There are lots of HA options. One that looks very nice to me (but that I
haven't tried) is DRDB which is a block level disk replication service. See
http://www.drbd.org/ for more information (and let us know how it looks).
The secondary nameserver may be of some help in recovery as well, but it is
unlikely to be as quick as a replicated disk and a CARP based IP address.
> If you can't run multiple namenodes, then that sort of implies the machine
> which is hosting *the* namenode needs to do all the traditional things to
> protect against data loss/corruption, including frequent backups, RAID
> mirroring, etc.
Some of these things are happening already, but the others are not a bad
idea at all. Consider your hardware carefully. RAID mirroring can
*decrease* reliability if you get a failure from either drive. Happened to
me on my home machine and I have heard of other cases as well. Even in
sophisticated implementations such as are done by Netapp, you can have drive
failures that freeze an entire shelf. My preference any more is replicated
simple machines rather than fancy machines.
Re: Multiple HDFS paths and Multiple Masters...
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 9/13/07 6:00 AM, "C G" <pa...@yahoo.com> wrote:
> I'd like to run nodes with around 2T of local disk set up as JBOD. So I
> would have 4 separate file systems per machine, for example /hdfs_a, /hdfs_b,
> /hdfs_c, /hdfs_d . Is it possible to configure things so that HDFS knows
> about all 4 file systems?
This is exactly how we configure all of our nodes. We have 4 drives
that have a chunk out of them for / and swap. The rest is dedicated to
hadoop, HDFS, etc.
> Since we're using HDFS replication I see no point in
> using RAID-anything...to me that's the whole point of replication Comments?
We haven't done any tests around what sort of performance difference a
stripe+concat configuration would make. In theory, it might be somewhat
faster since you have a lot more spindles. However, you also increase your
risk since a single drive failure would take out the entire data node rather
than just ~25% of it. As you point out, the replication factor makes RAID
fairly useless from a data recovery perspective.
> If you can't run multiple namenodes, then that sort of implies the machine
> which is hosting *the* namenode needs to do all the traditional things to
> protect against data loss/corruption, including frequent backups, RAID
> mirroring, etc.
We configure our primary namenode such that it has two copies of the
fsimage and edits file: one on the local machine and one on a NFS server.
If for some reason the local machine blows up, we can rebuild with the NFS
version.