Storing large number of files using hadoop

[Manoj Bist <ma...@gmail.com>]

Hi,

I need to serve a large number of audio/video files from web server.
The file size is in the range of few MB for each file.  These files
will be generated by a background process.
Each file is identified logically by a 'key' which looks like a url. I
am looking for a caching/storage solution for this.

1) Given a key, I should be able to stream the file to the user(web client)
2) Given a key, I should be able to write the content to the file.
3) Caching solution should take care of distributing the files across
different machines.

I am wondering if I can use Hadoop for this?

Thanx,
MB

Re: Storing large number of files using hadoop

[Matt Kent <ma...@gmail.com>]

MogileFS works well for the situation you described. You could also
try HBase, which is built on Hadoop but is still under development.
http://wiki.apache.org/lucene-hadoop/Hbase

On 8/19/07, Garth Patil <ga...@gmail.com> wrote:
> I haven't tried to use Hadoop for serving files like this, but I have
> used MogileFS, which was built for exactly the application you have in
> mind.
> http://www.danga.com/mogilefs/
> /Garth
>
> On 8/19/07, Manoj Bist <ma...@gmail.com> wrote:
> > Hi,
> >
> > I need to serve a large number of audio/video files from web server.
> > The file size is in the range of few MB for each file.  These files
> > will be generated by a background process.
> > Each file is identified logically by a 'key' which looks like a url. I
> > am looking for a caching/storage solution for this.
> >
> > 1) Given a key, I should be able to stream the file to the user(web client)
> > 2) Given a key, I should be able to write the content to the file.
> > 3) Caching solution should take care of distributing the files across
> > different machines.
> >
> > I am wondering if I can use Hadoop for this?
> >
> > Thanx,
> > MB
> >
>

Re: Storing large number of files using hadoop

[Garth Patil <ga...@gmail.com>]

I haven't tried to use Hadoop for serving files like this, but I have
used MogileFS, which was built for exactly the application you have in
mind.
http://www.danga.com/mogilefs/
/Garth

On 8/19/07, Manoj Bist <ma...@gmail.com> wrote:
> Hi,
>
> I need to serve a large number of audio/video files from web server.
> The file size is in the range of few MB for each file.  These files
> will be generated by a background process.
> Each file is identified logically by a 'key' which looks like a url. I
> am looking for a caching/storage solution for this.
>
> 1) Given a key, I should be able to stream the file to the user(web client)
> 2) Given a key, I should be able to write the content to the file.
> 3) Caching solution should take care of distributing the files across
> different machines.
>
> I am wondering if I can use Hadoop for this?
>
> Thanx,
> MB
>