You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Lance Boomerang <lr...@boomerang.com> on 2007/09/06 23:43:13 UTC
Anybody using HDFS as a long term storage solution?
Curious to see if anyone has been considering using HDFS for a general
storage platform.
I have been playing around with using HDFS to store video type assets.
wondering what type of mileage folks might be seeing?
Problems with too many small files?
Upgrading the system as new versions of HDFS/Hadoop come out.
Are there any recent updates on dealing with the single point of failure
in the single namenode?
Has anyone considered using DRBD to replicate the FsImage and
EditLog(s) and then use
Linux HA heartbeat to bring up the redundant node in the event of
failure? Also, has anyone
experimented with putting the alternate copies of the FsImage and
EditLog on a shared disk (NFS / NAS) ?
Another big question: Has anybody tried using HADOOP / HDFS across
multiple geographic sites?
Thanks in advance,
Lance
RE: Anybody using HDFS as a long term storage solution?
Posted by Dhruba Borthakur <dh...@yahoo-inc.com>.
The DataNode can be told which rack it lies in. This is documented in
http://issues.apache.org/jira/browse/HADOOP-692
thanks,
dhruba
-----Original Message-----
From: Torsten Curdt [mailto:tcurdt@apache.org]
Sent: Thursday, September 06, 2007 3:35 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Anybody using HDFS as a long term storage solution?
Is there a jira issue for it already?
Whom could I bug (bribe?) to document the rack-awareness feature? :)
cheers
--
Torsten
On 07.09.2007, at 00:26, Michael Bieniosek wrote:
> Well, there is an (undocumented?) way to get rack-awareness in the
> Datanode,
> so you could co-opt this to represent datacenter-awareness. I
> don't think
> there is such a rack-awareness ability for the DFSClient or
> TaskTracker
> though.
>
> -Michael
>
> On 9/6/07 3:10 PM, "Torsten Curdt" <tc...@apache.org> wrote:
>
>>> Another big question: Has anybody tried using HADOOP / HDFS across
>>> multiple geographic sites?
>>
>> That's actually a biggy I would be very much interested in, too
>>
>> cheers
>> --
>> Torsten
>
Re: Anybody using HDFS as a long term storage solution?
Posted by Michael Bieniosek <mi...@powerset.com>.
I don't know the answers to your questions. I learned what I know by
grepping through the DataNode.java source for "rack".
To use it, I set the "dfs.network.script" property in my hadoop config.
This contains the path to an executable which prints the name of the rack.
There's also a way to pass it on the command line to the DataNode class, so
it depends on how you start up your datanodes, I suppose.
-Michael
On 9/6/07 3:35 PM, "Torsten Curdt" <tc...@apache.org> wrote:
> Is there a jira issue for it already?
>
> Whom could I bug (bribe?) to document the rack-awareness feature? :)
>
> cheers
> --
> Torsten
>
> On 07.09.2007, at 00:26, Michael Bieniosek wrote:
>
>> Well, there is an (undocumented?) way to get rack-awareness in the
>> Datanode,
>> so you could co-opt this to represent datacenter-awareness. I
>> don't think
>> there is such a rack-awareness ability for the DFSClient or
>> TaskTracker
>> though.
>>
>> -Michael
>>
>> On 9/6/07 3:10 PM, "Torsten Curdt" <tc...@apache.org> wrote:
>>
>>>> Another big question: Has anybody tried using HADOOP / HDFS across
>>>> multiple geographic sites?
>>>
>>> That's actually a biggy I would be very much interested in, too
>>>
>>> cheers
>>> --
>>> Torsten
>>
>
Re: Anybody using HDFS as a long term storage solution?
Posted by Torsten Curdt <tc...@apache.org>.
Is there a jira issue for it already?
Whom could I bug (bribe?) to document the rack-awareness feature? :)
cheers
--
Torsten
On 07.09.2007, at 00:26, Michael Bieniosek wrote:
> Well, there is an (undocumented?) way to get rack-awareness in the
> Datanode,
> so you could co-opt this to represent datacenter-awareness. I
> don't think
> there is such a rack-awareness ability for the DFSClient or
> TaskTracker
> though.
>
> -Michael
>
> On 9/6/07 3:10 PM, "Torsten Curdt" <tc...@apache.org> wrote:
>
>>> Another big question: Has anybody tried using HADOOP / HDFS across
>>> multiple geographic sites?
>>
>> That's actually a biggy I would be very much interested in, too
>>
>> cheers
>> --
>> Torsten
>
Re: Anybody using HDFS as a long term storage solution?
Posted by Michael Bieniosek <mi...@powerset.com>.
Well, there is an (undocumented?) way to get rack-awareness in the Datanode,
so you could co-opt this to represent datacenter-awareness. I don't think
there is such a rack-awareness ability for the DFSClient or TaskTracker
though.
-Michael
On 9/6/07 3:10 PM, "Torsten Curdt" <tc...@apache.org> wrote:
>> Another big question: Has anybody tried using HADOOP / HDFS across
>> multiple geographic sites?
>
> That's actually a biggy I would be very much interested in, too
>
> cheers
> --
> Torsten
Re: Anybody using HDFS as a long term storage solution?
Posted by Torsten Curdt <tc...@apache.org>.
> Another big question: Has anybody tried using HADOOP / HDFS across
> multiple geographic sites?
That's actually a biggy I would be very much interested in, too
cheers
--
Torsten