You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Dongsheng Wang <ph...@yahoo.com> on 2007/09/05 22:03:16 UTC
Use HDFS as a long term storage solution?
We are looking at using HDFS as a long term storage solution. We want to use it to stored lots of files. The file could be big and small, they are images, videos etc... We only write the files once, and may read them many times. Sounds like it is perfect to use HDFS.
The concern is that since its been engineered to support MapReduce there may be fundamental assumptions that the data being stored by HDFS is transient in nature. Obviously for our scalable storage solution zero data loss or corruption is a heavy requirement.
Is anybody using HDFS as a long term storage solution? Interested in any info. Thanks
- ds
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
Re: Use HDFS as a long term storage solution?
Posted by Taeho Kang <tk...@gmail.com>.
Please also note that the current design only allows you up to a certain
number of files given a limited amount of RAM for namenode. The number
depends on a various factors (e.g. # of dirs, # of blocks, etc.) but a
through analysis found in
http://issues.apache.org/jira/browse/HADOOP-1687shows it's around 1M
files per 1G of RAM.
Of course, if you could have an infinite amount of RAM, then you could store
an infinite number of files.
As for the reliabilities of the HDFS, as long as you take care of your
namenode alright (e.g. your namenode being the most powerful and reliable
machine with a failover machine assigned to it), I find it shouldn't be a
much of a problem. I wouldn't use HDFS for any front end, time-critical
service, just yet, but I think it's good enough to use for back-end system (
e.g. log storage and analysis)
On 9/6/07, Dongsheng Wang <ph...@yahoo.com> wrote:
>
>
> We are looking at using HDFS as a long term storage solution. We want to
> use it to stored lots of files. The file could be big and small, they are
> images, videos etc... We only write the files once, and may read them many
> times. Sounds like it is perfect to use HDFS.
>
> The concern is that since it's been engineered to support MapReduce there
> may be fundamental assumptions that the data being stored by HDFS is
> transient in nature. Obviously for our scalable storage solution zero data
> loss or corruption is a heavy requirement.
>
> Is anybody using HDFS as a long term storage solution? Interested in any
> info. Thanks
>
> - ds
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
--
Taeho Kang [tkang.blogspot.com]
Software Engineer, NHN Corporation, Korea
Re: Use HDFS as a long term storage solution?
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hello,
It should be possible but I'm not currently doing so. I think this
method
is how one of the major video sites stores thumbnail images.
Maybe someone with more knowledge can help.
Regards,
Peter W.
Dongsheng Wang wrote:
> now sure about the advantage of using HBase over HDFS, can you give
> some insights on that?
>
> - dongsheng
>
> Peter W. wrote:
>
> Hello,
>
> Could images be turned into byte arrays and stored in HBase? Also,
> you will need a JVM that
> supports the -d64 flag otherwise you can only allocate up to 2048M to
> each machine.
>
> ...
>
>
> Dongsheng Wang wrote:
>
>>
>> We are looking at using HDFS as a long term storage solution. We
>> want to use it to stored lots of files. The file could be big and
>> small, they are images, videos etc... We only write the files once,
>> and may read them many times. Sounds like it is perfect to use HDFS.
Re: Use HDFS as a long term storage solution?
Posted by Dongsheng Wang <ph...@yahoo.com>.
sorry, I meant to see "not" sure
Dongsheng Wang <ph...@yahoo.com> wrote: now sure about the advantage of using HBase over HDFS, can you give some insights on that?
- dongsheng
"Peter W."
wrote: Hello,
Could images be turned into byte arrays and stored in HBase? Also,
you will need a JVM that
supports the -d64 flag otherwise you can only allocate up to 2048M to
each machine.
Peter W.
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
>
> We are looking at using HDFS as a long term storage solution. We
> want to use it to stored lots of files. The file could be big and
> small, they are images, videos etc... We only write the files once,
> and may read them many times. Sounds like it is perfect to use HDFS.
---------------------------------
Building a website is a piece of cake.
Yahoo! Small Business gives you all the tools to get online.
---------------------------------
Got a little couch potato?
Check out fun summer activities for kids.
Re: Use HDFS as a long term storage solution?
Posted by Ted Dunning <td...@veoh.com>.
Storing lots of small files would be a bad thing in HDFS. Storing lots of
small records in HBASE is a normal mode of operation.
On 9/6/07 11:25 AM, "Dongsheng Wang" <ph...@yahoo.com> wrote:
> not sure about the advantage of using HBase over HDFS, can you give some
> insights on that?
>
> - dongsheng
>
> "Peter W." <pe...@marketingbrokers.com> wrote: Hello,
>
> Could images be turned into byte arrays and stored in HBase? Also,
> you will need a JVM that
> supports the -d64 flag otherwise you can only allocate up to 2048M to
> each machine.
>
> Peter W.
>
>
> On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
>
>>
>> We are looking at using HDFS as a long term storage solution. We
>> want to use it to stored lots of files. The file could be big and
>> small, they are images, videos etc... We only write the files once,
>> and may read them many times. Sounds like it is perfect to use HDFS.
>
>
>
> ---------------------------------
> Building a website is a piece of cake.
> Yahoo! Small Business gives you all the tools to get online.
Re: Use HDFS as a long term storage solution?
Posted by Dongsheng Wang <ph...@yahoo.com>.
now sure about the advantage of using HBase over HDFS, can you give some insights on that?
- dongsheng
"Peter W." <pe...@marketingbrokers.com> wrote: Hello,
Could images be turned into byte arrays and stored in HBase? Also,
you will need a JVM that
supports the -d64 flag otherwise you can only allocate up to 2048M to
each machine.
Peter W.
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
>
> We are looking at using HDFS as a long term storage solution. We
> want to use it to stored lots of files. The file could be big and
> small, they are images, videos etc... We only write the files once,
> and may read them many times. Sounds like it is perfect to use HDFS.
---------------------------------
Building a website is a piece of cake.
Yahoo! Small Business gives you all the tools to get online.
Re: Use HDFS as a long term storage solution?
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hello,
Could images be turned into byte arrays and stored in HBase? Also,
you will need a JVM that
supports the -d64 flag otherwise you can only allocate up to 2048M to
each machine.
Peter W.
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
>
> We are looking at using HDFS as a long term storage solution. We
> want to use it to stored lots of files. The file could be big and
> small, they are images, videos etc... We only write the files once,
> and may read them many times. Sounds like it is perfect to use HDFS.
Re: Use HDFS as a long term storage solution?
Posted by Dongsheng Wang <ph...@yahoo.com>.
Thanks Owen,
I have some related questions, and think you might know the answers:
1. Any solution for namenode failure?
2. When snapshot will be impemented?
3. Is there any writting/reading throughput testing?
Dongsheng
Owen O'Malley <oo...@yahoo-inc.com> wrote:
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
> The concern is that since its been engineered to support MapReduce
> there may be fundamental assumptions that the data being stored by
> HDFS is transient in nature. Obviously for our scalable storage
> solution zero data loss or corruption is a heavy requirement.
Actually, HDFS was started before Map/Reduce. *smile* Map/Reduce does
not use HDFS for the transient storage, just the persistent storage
at the end of the job. HDFS should be a fairly good match for your
needs.
> Is anybody using HDFS as a long term storage solution? Interested
> in any info. Thanks
We have 1 petabyte of relatively persistent user data...
-- Owen
---------------------------------
Park yourself in front of a world of choices in alternative vehicles.
Visit the Yahoo! Auto Green Center.
Re: Use HDFS as a long term storage solution?
Posted by Owen O'Malley <oo...@yahoo-inc.com>.
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
> The concern is that since it’s been engineered to support MapReduce
> there may be fundamental assumptions that the data being stored by
> HDFS is transient in nature. Obviously for our scalable storage
> solution zero data loss or corruption is a heavy requirement.
Actually, HDFS was started before Map/Reduce. *smile* Map/Reduce does
not use HDFS for the transient storage, just the persistent storage
at the end of the job. HDFS should be a fairly good match for your
needs.
> Is anybody using HDFS as a long term storage solution? Interested
> in any info. Thanks
We have 1 petabyte of relatively persistent user data...
-- Owen
Re: Use HDFS as a long term storage solution?
Posted by Dongsheng Wang <ph...@yahoo.com>.
2) loading lots of small files into HDFS, causing it to hang on a Map/Reduce
job and subsequently display corruption on restart;
I had problem too when we load thousands files into hadoop and run MapReduce job. I thought it is just a performance problem. And, It apparently Hadoop start a new process for each task. I believe that might be the reason.
Jeff Hammerbacher <je...@gmail.com> wrote: We have very similar plans for Hadoop to what C G quotes below, but we've
found the stability of HDFS to be quite troublesome. We've corrupted HDFS
three different ways in a few weeks: 1) running jStack on the Namenode; 2)
loading lots of small files into HDFS, causing it to hang on a Map/Reduce
job and subsequently display corruption on restart; 3) upgrading to a newer
version of Hadoop. Thus we are very uncertain about treating HDFS as a
reliable long-term data store.
That being said, we're excited about the opportunities created by Hadoop so
we're going to put some time into making it more reliable and creating a
utility to archive data out of HDFS for backup purposes.
On 9/5/07, C G
wrote:
>
> Our intention is to use HDFS as the core of a large "data repository". We
> store "raw" data within HDFS on a more-or-less permanent basis, and
> map/reduce it to produce load files for our data warehouse. We have other
> plans as well all centered around storing data on a very long term basis in
> HDFS. So you're in good company...
>
> Our plan is for a 64T HDFS repository, with a replication factor of 3
> for a ~21T data space.
>
> C G
>
>
> Dongsheng Wang
wrote:
>
> We are looking at using HDFS as a long term storage solution. We want to
> use it to stored lots of files. The file could be big and small, they are
> images, videos etc... We only write the files once, and may read them many
> times. Sounds like it is perfect to use HDFS.
>
> The concern is that since it's been engineered to support MapReduce there
> may be fundamental assumptions that the data being stored by HDFS is
> transient in nature. Obviously for our scalable storage solution zero data
> loss or corruption is a heavy requirement.
>
> Is anybody using HDFS as a long term storage solution? Interested in any
> info. Thanks
>
> - ds
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
>
>
> ---------------------------------
> Ready for the edge of your seat? Check out tonight's top picks on Yahoo!
> TV.
---------------------------------
Building a website is a piece of cake.
Yahoo! Small Business gives you all the tools to get online.
Re: Use HDFS as a long term storage solution?
Posted by Eric Baldeschwieler <er...@yahoo-inc.com>.
We are very interested in ideas and patches to improve the systems
stability.
This is very young software, but we are using it at very large scale
and intend to keep enhancing it. We currently have a 2000 node file
system with 3TB raw storage per node and are supporting millions of
files.
With careful shepherding we've had no data loss events this year (we
only recently reach 2000 nodes). The biggest risks right now are
user errors, due to lack of basic access protections. Some fixes for
that are in the works.
On Sep 5, 2007, at 5:56 PM, Jeff Hammerbacher wrote:
> We have very similar plans for Hadoop to what C G quotes below, but
> we've
> found the stability of HDFS to be quite troublesome. We've
> corrupted HDFS
> three different ways in a few weeks: 1) running jStack on the
> Namenode; 2)
> loading lots of small files into HDFS, causing it to hang on a Map/
> Reduce
> job and subsequently display corruption on restart; 3) upgrading to
> a newer
> version of Hadoop. Thus we are very uncertain about treating HDFS
> as a
> reliable long-term data store.
>
> That being said, we're excited about the opportunities created by
> Hadoop so
> we're going to put some time into making it more reliable and
> creating a
> utility to archive data out of HDFS for backup purposes.
>
> On 9/5/07, C G <pa...@yahoo.com> wrote:
> >
> > Our intention is to use HDFS as the core of a large "data
> repository". We
> > store "raw" data within HDFS on a more-or-less permanent basis, and
> > map/reduce it to produce load files for our data warehouse. We
> have other
> > plans as well all centered around storing data on a very long
> term basis in
> > HDFS. So you're in good company...
> >
> > Our plan is for a 64T HDFS repository, with a replication
> factor of 3
> > for a ~21T data space.
> >
> > C G
> >
> >
> > Dongsheng Wang <ph...@yahoo.com> wrote:
> >
> > We are looking at using HDFS as a long term storage solution. We
> want to
> > use it to stored lots of files. The file could be big and small,
> they are
> > images, videos etc... We only write the files once, and may read
> them many
> > times. Sounds like it is perfect to use HDFS.
> >
> > The concern is that since it's been engineered to support
> MapReduce there
> > may be fundamental assumptions that the data being stored by HDFS is
> > transient in nature. Obviously for our scalable storage solution
> zero data
> > loss or corruption is a heavy requirement.
> >
> > Is anybody using HDFS as a long term storage solution? Interested
> in any
> > info. Thanks
> >
> > - ds
> >
> >
> > ---------------------------------
> > Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> > links.
> >
> >
> > ---------------------------------
> > Ready for the edge of your seat? Check out tonight's top picks on
> Yahoo!
> > TV.
>
Re: Use HDFS as a long term storage solution?
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 9/5/07 5:56 PM, "Jeff Hammerbacher" <je...@gmail.com> wrote:
> 3) upgrading to a newer version of Hadoop.
I'm hoping at some point to go through what is on the Hadoop wiki and
compare it to the process that we've built up here. I'm pretty sure we're
doing a lot more safe guarding/checking of data pre- and post- upgrade that
either aren't part of the documented procedure or are marked as optional
that we treat as required.
Re: Use HDFS as a long term storage solution?
Posted by Jeff Hammerbacher <je...@gmail.com>.
We have very similar plans for Hadoop to what C G quotes below, but we've
found the stability of HDFS to be quite troublesome. We've corrupted HDFS
three different ways in a few weeks: 1) running jStack on the Namenode; 2)
loading lots of small files into HDFS, causing it to hang on a Map/Reduce
job and subsequently display corruption on restart; 3) upgrading to a newer
version of Hadoop. Thus we are very uncertain about treating HDFS as a
reliable long-term data store.
That being said, we're excited about the opportunities created by Hadoop so
we're going to put some time into making it more reliable and creating a
utility to archive data out of HDFS for backup purposes.
On 9/5/07, C G <pa...@yahoo.com> wrote:
>
> Our intention is to use HDFS as the core of a large "data repository". We
> store "raw" data within HDFS on a more-or-less permanent basis, and
> map/reduce it to produce load files for our data warehouse. We have other
> plans as well all centered around storing data on a very long term basis in
> HDFS. So you're in good company...
>
> Our plan is for a 64T HDFS repository, with a replication factor of 3
> for a ~21T data space.
>
> C G
>
>
> Dongsheng Wang <ph...@yahoo.com> wrote:
>
> We are looking at using HDFS as a long term storage solution. We want to
> use it to stored lots of files. The file could be big and small, they are
> images, videos etc... We only write the files once, and may read them many
> times. Sounds like it is perfect to use HDFS.
>
> The concern is that since it's been engineered to support MapReduce there
> may be fundamental assumptions that the data being stored by HDFS is
> transient in nature. Obviously for our scalable storage solution zero data
> loss or corruption is a heavy requirement.
>
> Is anybody using HDFS as a long term storage solution? Interested in any
> info. Thanks
>
> - ds
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web
> links.
>
>
> ---------------------------------
> Ready for the edge of your seat? Check out tonight's top picks on Yahoo!
> TV.
Re: Use HDFS as a long term storage solution?
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hello,
Let me clarify. The Hadoop software works
flawlessly out of the box but those of us
without a cluster (few machines) need a way
to maximize hardware resources.
My machine is 64-bit and a single-node
Hadoop job can take up to 1GB resident
private memory (RPRVT) on a one box job.
Possibly this is because I do String
concatenations within a single reduce instead
of running multiple identity mappers and
additional reduce passes.
Now, I'd like to run HBase on a single node pulling
megabytes of data in and out and imagine that memory
resources would be quickly exhausted. The idea was
to grow the JVM by using -Xms1g -Xmx4g.
Since I'd probably need to run solaris to try this
the regular Hadoop configuration should be fine.
Peter W.
On Sep 7, 2007, at 10:28 AM, Peter W. wrote:
> Lance,
>
> Allocating more than 2GB to the JVM requires 64-bit.
>
> Regards,
>
> Peter W.
>
>
> Peter W. wrote:
>
>> Hi,
>>
>> My only experience is using Hadoop and HBase shell on OSX Tiger.
>>
>> It seems that if you are going to store large amounts of data
>> w/HBase allocating more than 2GB to the JVM would be helpful.
>>
>> Not sure about about your HDFS compatibility questions.
>>
>> Regards,
>>
>> Peter W.
>>
>> Lance Boomerang wrote:
>>
>>> So are we assuming that all the HDFS nodes are on 64bit hosts?
>>>
>>> Any idea if using the Hadoop Java API on 32bit clients would work
>>> to read / write files into a 64 bit HDFS setup over the network
>>> (HTTP)?
>>>
>>> Thanks,
>>>
>>> Lance
>
Re: Use HDFS as a long term storage solution?
Posted by "Peter W." <pe...@marketingbrokers.com>.
Lance,
Allocating more than 2GB to the JVM requires 64-bit.
Regards,
Peter W.
Peter W. wrote:
> Hi,
>
> My only experience is using Hadoop and HBase shell on OSX Tiger.
>
> It seems that if you are going to store large amounts of data
> w/HBase allocating more than 2GB to the JVM would be helpful.
>
> Not sure about about your HDFS compatibility questions.
>
> Regards,
>
> Peter W.
>
> Lance Boomerang wrote:
>
>> So are we assuming that all the HDFS nodes are on 64bit hosts?
>>
>> Any idea if using the Hadoop Java API on 32bit clients would work
>> to read / write files into a 64 bit HDFS setup over the network
>> (HTTP)?
>>
>> Thanks,
>>
>> Lance
Re: Use HDFS as a long term storage solution?
Posted by "Peter W." <pe...@marketingbrokers.com>.
Hi,
My only experience is using Hadoop and HBase shell on OSX Tiger.
It seems that if you are going to store large amounts of data
w/HBase allocating more than 2GB to the JVM would be helpful.
Not sure about about your HDFS compatibility questions.
Regards,
Peter W.
Lance Boomerang wrote:
> So are we assuming that all the HDFS nodes are on 64bit hosts?
>
> Any idea if using the Hadoop Java API on 32bit clients would work
> to read / write files into a 64 bit HDFS setup over the network
> (HTTP)?
>
> Thanks,
>
> Lance
Re: Use HDFS as a long term storage solution?
Posted by Lance Boomerang <lr...@boomerang.com>.
So are we assuming that all the HDFS nodes are on 64bit hosts?
Any idea if using the Hadoop Java API on 32bit clients would work to
read / write files into a 64 bit HDFS setup over the network (HTTP)?
Thanks,
Lance
----------------------------
"Could images be turned into byte arrays and stored in HBase? Also, you
will need a JVM that supports the -d64 flag otherwise you can only
allocate up to 2048M to each machine.
Peter W."
On Sep 5, 2007, at 1:03 PM, Dongsheng Wang wrote:
We are looking at using HDFS as a long term storage solution. We
want to use it to stored lots of files. The file could be big and
small, they are images, videos etc... We only write the files once,
and may read them many times. Sounds like it is perfect to use HDFS.
RE: Use HDFS as a long term storage solution?
Posted by C G <pa...@yahoo.com>.
http://research.yahoo.com/project/pig
Looks very cool on paper...I haven't been able to get it running yet...
"Earney, Billy C." <ea...@umsystem.edu> wrote:
Could someone explain what Pig is? (ie, Ted said ' That storage format
would allow you to use Pig. Pig is, unfortunately, still quite limited
in that input data must be fielded and in a simple format.')
Thanks!
-----Original Message-----
From: Ted Dunning [mailto:tdunning@veoh.com]
Sent: Thursday, September 06, 2007 9:26 PM
To: hadoop-user@lucene.apache.org; hadoop-user@lucene.apache.org
Subject: RE: Use HDFS as a long term storage solution?
That size and number of files is fine for storage in HDFS. Your
situation is comparable to mine except that I collect 2-5 files per hour
and these files are slightly larger. My files are in a compressed and
encrypted format, but keeping them in a SequenceFile compressed format
would make map/reduce noticeably more efficient. This is because the
file splits can be arranged by the JobTracker to coincide with the disk
blocks in HDFS. That can result significantly higher percentage of
tasks working against local inputs.
Note that at these accumulation rates, you are really only talking about
< 100K files over a year. That still counts as a small number of files.
A large number of files is 10M or more.
You might also be well served if you were to keep your data in a (block
compressed) tab-delimited form even at the cost of some grotesqueness.
That storage format would allow you to use Pig. Pig is, unfortunately,
still quite limited in that input data must be fielded and in a simple
format.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thu 9/6/2007 6:30 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Use HDFS as a long term storage solution?
Right, my preference would be to use HDFS exclusively...except that
there are potential issues with many small files in HDFS and a
suggestion that perhaps MogileFS might be better for many small files.
My strong preference is to store everything in HDFS, then do map/reduce
with the small files to produce results. Since there is a concern about
storing a lot of small files in HDFS, I now wonder if I should collect
small files into MogileFS, the periodically merge them together to
create large files, and then store those in HDFS and then issue my
map/reduces. Ick, that sounds complex/time-consuming just writing about
it :-(.
The files I anticipate processing are all compressed (gzip), and are
on the order of 80-200M compressed. I expect to collect 4-8 of these
files per hour for most hours in the day.
---------------------------------
Need a vacation? Get great deals to amazing places on Yahoo! Travel.
RE: Use HDFS as a long term storage solution?
Posted by "Earney, Billy C." <ea...@umsystem.edu>.
Could someone explain what Pig is? (ie, Ted said ' That storage format
would allow you to use Pig. Pig is, unfortunately, still quite limited
in that input data must be fielded and in a simple format.')
Thanks!
-----Original Message-----
From: Ted Dunning [mailto:tdunning@veoh.com]
Sent: Thursday, September 06, 2007 9:26 PM
To: hadoop-user@lucene.apache.org; hadoop-user@lucene.apache.org
Subject: RE: Use HDFS as a long term storage solution?
That size and number of files is fine for storage in HDFS. Your
situation is comparable to mine except that I collect 2-5 files per hour
and these files are slightly larger. My files are in a compressed and
encrypted format, but keeping them in a SequenceFile compressed format
would make map/reduce noticeably more efficient. This is because the
file splits can be arranged by the JobTracker to coincide with the disk
blocks in HDFS. That can result significantly higher percentage of
tasks working against local inputs.
Note that at these accumulation rates, you are really only talking about
< 100K files over a year. That still counts as a small number of files.
A large number of files is 10M or more.
You might also be well served if you were to keep your data in a (block
compressed) tab-delimited form even at the cost of some grotesqueness.
That storage format would allow you to use Pig. Pig is, unfortunately,
still quite limited in that input data must be fielded and in a simple
format.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thu 9/6/2007 6:30 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Use HDFS as a long term storage solution?
Right, my preference would be to use HDFS exclusively...except that
there are potential issues with many small files in HDFS and a
suggestion that perhaps MogileFS might be better for many small files.
My strong preference is to store everything in HDFS, then do map/reduce
with the small files to produce results. Since there is a concern about
storing a lot of small files in HDFS, I now wonder if I should collect
small files into MogileFS, the periodically merge them together to
create large files, and then store those in HDFS and then issue my
map/reduces. Ick, that sounds complex/time-consuming just writing about
it :-(.
The files I anticipate processing are all compressed (gzip), and are
on the order of 80-200M compressed. I expect to collect 4-8 of these
files per hour for most hours in the day.
RE: Use HDFS as a long term storage solution?
Posted by Ted Dunning <td...@veoh.com>.
That size and number of files is fine for storage in HDFS. Your situation is comparable to mine except that I collect 2-5 files per hour and these files are slightly larger. My files are in a compressed and encrypted format, but keeping them in a SequenceFile compressed format would make map/reduce noticeably more efficient. This is because the file splits can be arranged by the JobTracker to coincide with the disk blocks in HDFS. That can result significantly higher percentage of tasks working against local inputs.
Note that at these accumulation rates, you are really only talking about < 100K files over a year. That still counts as a small number of files. A large number of files is 10M or more.
You might also be well served if you were to keep your data in a (block compressed) tab-delimited form even at the cost of some grotesqueness. That storage format would allow you to use Pig. Pig is, unfortunately, still quite limited in that input data must be fielded and in a simple format.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thu 9/6/2007 6:30 PM
To: hadoop-user@lucene.apache.org
Subject: RE: Use HDFS as a long term storage solution?
Right, my preference would be to use HDFS exclusively...except that there are potential issues with many small files in HDFS and a suggestion that perhaps MogileFS might be better for many small files. My strong preference is to store everything in HDFS, then do map/reduce with the small files to produce results. Since there is a concern about storing a lot of small files in HDFS, I now wonder if I should collect small files into MogileFS, the periodically merge them together to create large files, and then store those in HDFS and then issue my map/reduces. Ick, that sounds complex/time-consuming just writing about it :-(.
The files I anticipate processing are all compressed (gzip), and are on the order of 80-200M compressed. I expect to collect 4-8 of these files per hour for most hours in the day.
RE: Use HDFS as a long term storage solution?
Posted by C G <pa...@yahoo.com>.
Right, my preference would be to use HDFS exclusively...except that there are potential issues with many small files in HDFS and a suggestion that perhaps MogileFS might be better for many small files. My strong preference is to store everything in HDFS, then do map/reduce with the small files to produce results. Since there is a concern about storing a lot of small files in HDFS, I now wonder if I should collect small files into MogileFS, the periodically merge them together to create large files, and then store those in HDFS and then issue my map/reduces. Ick, that sounds complex/time-consuming just writing about it :-(.
The files I anticipate processing are all compressed (gzip), and are on the order of 80-200M compressed. I expect to collect 4-8 of these files per hour for most hours in the day.
Ted Dunning <td...@veoh.com> wrote:
Absolutely, you can use map/reduce without HDFS.
That is the standard debugging style, for one thing. For another, Nutch is all about accessing non-HDFS data (i.e. the web).
If I get your drift this time, though, I would expect that you will have severe problems with bandwidth if you have a lot of task nodes working on a conventional file store. One of the great virtues of map/reduce + HDFS is that most of the map inputs are read from local disk, as are the reduce inputs. Any system that doesn't coordinate work with storage this way is likely to suffer from lower througput due to network congestion.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thu 9/6/2007 5:54 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Use HDFS as a long term storage solution?
Actually my question is 'Can you use Map/Reduce without HDFS?"
Ted Dunning wrote:
You can definitely use HDFS without map/reduce. It should be pretty easy to
use it from a variety of languages as well, although it is unlikely that
there are language bindings available off the shelf.
....
On 9/6/07 1:04 PM, "C G"
wrote:
> Do you have to use HDFS with map/reduce? I don't fully understand how closely
> bound map/reduce is to HDFS.
>
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
RE: Use HDFS as a long term storage solution?
Posted by Ted Dunning <td...@veoh.com>.
Absolutely, you can use map/reduce without HDFS.
That is the standard debugging style, for one thing. For another, Nutch is all about accessing non-HDFS data (i.e. the web).
If I get your drift this time, though, I would expect that you will have severe problems with bandwidth if you have a lot of task nodes working on a conventional file store. One of the great virtues of map/reduce + HDFS is that most of the map inputs are read from local disk, as are the reduce inputs. Any system that doesn't coordinate work with storage this way is likely to suffer from lower througput due to network congestion.
-----Original Message-----
From: C G [mailto:parallelguy@yahoo.com]
Sent: Thu 9/6/2007 5:54 PM
To: hadoop-user@lucene.apache.org
Subject: Re: Use HDFS as a long term storage solution?
Actually my question is 'Can you use Map/Reduce without HDFS?"
Ted Dunning <td...@veoh.com> wrote:
You can definitely use HDFS without map/reduce. It should be pretty easy to
use it from a variety of languages as well, although it is unlikely that
there are language bindings available off the shelf.
...
On 9/6/07 1:04 PM, "C G"
wrote:
> Do you have to use HDFS with map/reduce? I don't fully understand how closely
> bound map/reduce is to HDFS.
>
Re: Use HDFS as a long term storage solution?
Posted by C G <pa...@yahoo.com>.
Actually my question is 'Can you use Map/Reduce without HDFS?"
Ted Dunning <td...@veoh.com> wrote:
You can definitely use HDFS without map/reduce. It should be pretty easy to
use it from a variety of languages as well, although it is unlikely that
there are language bindings available off the shelf.
It is very frustrating to have two kinds of data store, but if you are doing
two kinds of thing, it might just be justified. If you look at the
different priorities of the team, you will have a pretty simple inference as
to what the strengths of the platforms will be. With Mogile, the goal is
serving content sites with high reliability from a variety of languages
without implementing anything more complicated than needed. They ignore
issues like how to place tasks to process data efficiently. With HDFS, the
goal is to facilitate large-scale computation. The result is much less
focused on the operational issues that will arise if you are trying to keep
data alive for years at a time.
On 9/6/07 1:04 PM, "C G"
wrote:
> Do you have to use HDFS with map/reduce? I don't fully understand how closely
> bound map/reduce is to HDFS.
>
> In our application it might make more sense to accrue data using MogileFS
> and place post-processed data (i.e. larger data) into HDFS for additional
> processing.
>
> Comments?
>
> Ted Dunning wrote:
>
> Hadoop may not be what you want for storing lots and lots of files.
>
> If you need to store >10^7 files or if you are storing lots of small (<40MB)
> files, then you may prefer a solution like mogileFS. It is engineered for a
> very different purpose than hadoop, but may be more appropriate for what you
> want. It is also already intended for web-scale reliable applications so
> there is a bit more that you can do for redundancy.
>
> On the other hand, HDFS might be just what you need.
>
>
> On 9/5/07 1:03 PM, "Dongsheng Wang"
> wrote:
>
>>
>> We are looking at using HDFS as a long term storage solution. We want to use
>> it to stored lots of files. The file could be big and small, they are images,
>> videos etc... We only write the files once, and may read them many times.
>> Sounds like it is perfect to use HDFS.
>>
>> The concern is that since it¹s been engineered to support MapReduce there may
>> be fundamental assumptions that the data being stored by HDFS is transient in
>> nature. Obviously for our scalable storage solution zero data loss or
>> corruption is a heavy requirement.
>>
>> Is anybody using HDFS as a long term storage solution? Interested in any
>> info.
>> Thanks
>>
>> - ds
>>
>>
>> ---------------------------------
>> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
>
>
>
>
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who
> knows.
> Yahoo! Answers - Check it out.
---------------------------------
Pinpoint customers who are looking for what you sell.
Re: Use HDFS as a long term storage solution?
Posted by Ted Dunning <td...@veoh.com>.
You can definitely use HDFS without map/reduce. It should be pretty easy to
use it from a variety of languages as well, although it is unlikely that
there are language bindings available off the shelf.
It is very frustrating to have two kinds of data store, but if you are doing
two kinds of thing, it might just be justified. If you look at the
different priorities of the team, you will have a pretty simple inference as
to what the strengths of the platforms will be. With Mogile, the goal is
serving content sites with high reliability from a variety of languages
without implementing anything more complicated than needed. They ignore
issues like how to place tasks to process data efficiently. With HDFS, the
goal is to facilitate large-scale computation. The result is much less
focused on the operational issues that will arise if you are trying to keep
data alive for years at a time.
On 9/6/07 1:04 PM, "C G" <pa...@yahoo.com> wrote:
> Do you have to use HDFS with map/reduce? I don't fully understand how closely
> bound map/reduce is to HDFS.
>
> In our application it might make more sense to accrue data using MogileFS
> and place post-processed data (i.e. larger data) into HDFS for additional
> processing.
>
> Comments?
>
> Ted Dunning <td...@veoh.com> wrote:
>
> Hadoop may not be what you want for storing lots and lots of files.
>
> If you need to store >10^7 files or if you are storing lots of small (<40MB)
> files, then you may prefer a solution like mogileFS. It is engineered for a
> very different purpose than hadoop, but may be more appropriate for what you
> want. It is also already intended for web-scale reliable applications so
> there is a bit more that you can do for redundancy.
>
> On the other hand, HDFS might be just what you need.
>
>
> On 9/5/07 1:03 PM, "Dongsheng Wang"
> wrote:
>
>>
>> We are looking at using HDFS as a long term storage solution. We want to use
>> it to stored lots of files. The file could be big and small, they are images,
>> videos etc... We only write the files once, and may read them many times.
>> Sounds like it is perfect to use HDFS.
>>
>> The concern is that since it¹s been engineered to support MapReduce there may
>> be fundamental assumptions that the data being stored by HDFS is transient in
>> nature. Obviously for our scalable storage solution zero data loss or
>> corruption is a heavy requirement.
>>
>> Is anybody using HDFS as a long term storage solution? Interested in any
>> info.
>> Thanks
>>
>> - ds
>>
>>
>> ---------------------------------
>> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
>
>
>
>
> ---------------------------------
> Be a better Heartthrob. Get better relationship answers from someone who
> knows.
> Yahoo! Answers - Check it out.
Re: Use HDFS as a long term storage solution?
Posted by C G <pa...@yahoo.com>.
Do you have to use HDFS with map/reduce? I don't fully understand how closely bound map/reduce is to HDFS.
In our application it might make more sense to accrue data using MogileFS and place post-processed data (i.e. larger data) into HDFS for additional processing.
Comments?
Ted Dunning <td...@veoh.com> wrote:
Hadoop may not be what you want for storing lots and lots of files.
If you need to store >10^7 files or if you are storing lots of small (<40MB)
files, then you may prefer a solution like mogileFS. It is engineered for a
very different purpose than hadoop, but may be more appropriate for what you
want. It is also already intended for web-scale reliable applications so
there is a bit more that you can do for redundancy.
On the other hand, HDFS might be just what you need.
On 9/5/07 1:03 PM, "Dongsheng Wang"
wrote:
>
> We are looking at using HDFS as a long term storage solution. We want to use
> it to stored lots of files. The file could be big and small, they are images,
> videos etc... We only write the files once, and may read them many times.
> Sounds like it is perfect to use HDFS.
>
> The concern is that since it¹s been engineered to support MapReduce there may
> be fundamental assumptions that the data being stored by HDFS is transient in
> nature. Obviously for our scalable storage solution zero data loss or
> corruption is a heavy requirement.
>
> Is anybody using HDFS as a long term storage solution? Interested in any info.
> Thanks
>
> - ds
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
---------------------------------
Be a better Heartthrob. Get better relationship answers from someone who knows.
Yahoo! Answers - Check it out.
Re: Use HDFS as a long term storage solution?
Posted by Ted Dunning <td...@veoh.com>.
Hadoop may not be what you want for storing lots and lots of files.
If you need to store >10^7 files or if you are storing lots of small (<40MB)
files, then you may prefer a solution like mogileFS. It is engineered for a
very different purpose than hadoop, but may be more appropriate for what you
want. It is also already intended for web-scale reliable applications so
there is a bit more that you can do for redundancy.
On the other hand, HDFS might be just what you need.
On 9/5/07 1:03 PM, "Dongsheng Wang" <ph...@yahoo.com> wrote:
>
> We are looking at using HDFS as a long term storage solution. We want to use
> it to stored lots of files. The file could be big and small, they are images,
> videos etc... We only write the files once, and may read them many times.
> Sounds like it is perfect to use HDFS.
>
> The concern is that since it¹s been engineered to support MapReduce there may
> be fundamental assumptions that the data being stored by HDFS is transient in
> nature. Obviously for our scalable storage solution zero data loss or
> corruption is a heavy requirement.
>
> Is anybody using HDFS as a long term storage solution? Interested in any info.
> Thanks
>
> - ds
>
>
> ---------------------------------
> Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
Re: Use HDFS as a long term storage solution?
Posted by C G <pa...@yahoo.com>.
Our intention is to use HDFS as the core of a large "data repository". We store "raw" data within HDFS on a more-or-less permanent basis, and map/reduce it to produce load files for our data warehouse. We have other plans as well all centered around storing data on a very long term basis in HDFS. So you're in good company...
Our plan is for a 64T HDFS repository, with a replication factor of 3 for a ~21T data space.
C G
Dongsheng Wang <ph...@yahoo.com> wrote:
We are looking at using HDFS as a long term storage solution. We want to use it to stored lots of files. The file could be big and small, they are images, videos etc... We only write the files once, and may read them many times. Sounds like it is perfect to use HDFS.
The concern is that since its been engineered to support MapReduce there may be fundamental assumptions that the data being stored by HDFS is transient in nature. Obviously for our scalable storage solution zero data loss or corruption is a heavy requirement.
Is anybody using HDFS as a long term storage solution? Interested in any info. Thanks
- ds
---------------------------------
Yahoo! oneSearch: Finally, mobile search that gives answers, not web links.
---------------------------------
Ready for the edge of your seat? Check out tonight's top picks on Yahoo! TV.