You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Ananth Sarathy <an...@gmail.com> on 2010/10/26 18:28:31 UTC
Small File Management
I was wondering if there were any projects out there doing a small file
management layer on top of Hadoop? I know that HDFS is primarily for
map/reduce but I think companies are going to start using hdfs clusters as
storage in the cloud, and i was wondering if any work had been done on this.
Ananth T Sarathy
Re: Small File Management
Posted by Allen Wittenauer <aw...@linkedin.com>.
On Oct 26, 2010, at 9:28 AM, Ananth Sarathy wrote:
> I was wondering if there were any projects out there doing a small file
> management layer on top of Hadoop? I know that HDFS is primarily for
> map/reduce but I think companies are going to start using hdfs clusters as
> storage in the cloud, and i was wondering if any work had been done on this.
> Ananth T Sarathy
http://hadoop.apache.org/common/docs/r0.20.2/hadoop_archives.html
Re: Small File Management
Posted by Todd Lipcon <to...@cloudera.com>.
It's worth checking out the "har" tool as well.
I would say that HBase is good fit for binaries so long as the binaries
aren't huge. Anything under a few MB should be fine.
-Todd
On Tue, Oct 26, 2010 at 10:56 AM, Ananth Sarathy <ananth.t.sarathy@gmail.com
> wrote:
> Thanks, but that more of a one time use, not ongoing management.
> Ananth T Sarathy
>
>
> On Tue, Oct 26, 2010 at 12:31 PM, Mark Kerzner <markkerzner@gmail.com
> >wrote:
>
> > http://stuartsierra.com/2008/04/24/a-million-little-files
> >
> > On Tue, Oct 26, 2010 at 11:28 AM, Ananth Sarathy <
> > ananth.t.sarathy@gmail.com
> > > wrote:
> >
> > > I was wondering if there were any projects out there doing a small file
> > > management layer on top of Hadoop? I know that HDFS is primarily for
> > > map/reduce but I think companies are going to start using hdfs clusters
> > as
> > > storage in the cloud, and i was wondering if any work had been done on
> > > this.
> > > Ananth T Sarathy
> > >
> >
>
--
Todd Lipcon
Software Engineer, Cloudera
Re: Small File Management
Posted by Ananth Sarathy <an...@gmail.com>.
Thanks, but that more of a one time use, not ongoing management.
Ananth T Sarathy
On Tue, Oct 26, 2010 at 12:31 PM, Mark Kerzner <ma...@gmail.com>wrote:
> http://stuartsierra.com/2008/04/24/a-million-little-files
>
> On Tue, Oct 26, 2010 at 11:28 AM, Ananth Sarathy <
> ananth.t.sarathy@gmail.com
> > wrote:
>
> > I was wondering if there were any projects out there doing a small file
> > management layer on top of Hadoop? I know that HDFS is primarily for
> > map/reduce but I think companies are going to start using hdfs clusters
> as
> > storage in the cloud, and i was wondering if any work had been done on
> > this.
> > Ananth T Sarathy
> >
>
Re: Small File Management
Posted by Mark Kerzner <ma...@gmail.com>.
http://stuartsierra.com/2008/04/24/a-million-little-files
On Tue, Oct 26, 2010 at 11:28 AM, Ananth Sarathy <ananth.t.sarathy@gmail.com
> wrote:
> I was wondering if there were any projects out there doing a small file
> management layer on top of Hadoop? I know that HDFS is primarily for
> map/reduce but I think companies are going to start using hdfs clusters as
> storage in the cloud, and i was wondering if any work had been done on
> this.
> Ananth T Sarathy
>
Re: Small File Management
Posted by Fernando Padilla <fe...@alum.mit.edu>.
really, they are "adament" about not using it for binaries? I guess it
really depends on the size of the binaries. If you already have
experience with HBase, why don't you try it out and benchmark it?
On 10/26/10 10:55 AM, Ananth Sarathy wrote:
> Yeah I had looked into hbase, but they are pretty adament about not using it
> for binaries. We use hbase for other stuff, so that would have been our
> preference I know that BigTable serves some image tiles for google maps,
> but images tiles are a lot smaller in general.
> Ananth T Sarathy
>
>
> On Tue, Oct 26, 2010 at 1:07 PM, Patrick Angeles<pa...@cloudera.com>wrote:
>
>> HBase might fit the bill.
>>
>> On Tue, Oct 26, 2010 at 12:28 PM, Ananth Sarathy<
>> ananth.t.sarathy@gmail.com
>>> wrote:
>>> I was wondering if there were any projects out there doing a small file
>>> management layer on top of Hadoop? I know that HDFS is primarily for
>>> map/reduce but I think companies are going to start using hdfs clusters
>> as
>>> storage in the cloud, and i was wondering if any work had been done on
>>> this.
>>> Ananth T Sarathy
>>>
Re: Small File Management
Posted by Ananth Sarathy <an...@gmail.com>.
Yeah I had looked into hbase, but they are pretty adament about not using it
for binaries. We use hbase for other stuff, so that would have been our
preference I know that BigTable serves some image tiles for google maps,
but images tiles are a lot smaller in general.
Ananth T Sarathy
On Tue, Oct 26, 2010 at 1:07 PM, Patrick Angeles <pa...@cloudera.com>wrote:
> HBase might fit the bill.
>
> On Tue, Oct 26, 2010 at 12:28 PM, Ananth Sarathy <
> ananth.t.sarathy@gmail.com
> > wrote:
>
> > I was wondering if there were any projects out there doing a small file
> > management layer on top of Hadoop? I know that HDFS is primarily for
> > map/reduce but I think companies are going to start using hdfs clusters
> as
> > storage in the cloud, and i was wondering if any work had been done on
> > this.
> > Ananth T Sarathy
> >
>
Re: Small File Management
Posted by Patrick Angeles <pa...@cloudera.com>.
HBase might fit the bill.
On Tue, Oct 26, 2010 at 12:28 PM, Ananth Sarathy <ananth.t.sarathy@gmail.com
> wrote:
> I was wondering if there were any projects out there doing a small file
> management layer on top of Hadoop? I know that HDFS is primarily for
> map/reduce but I think companies are going to start using hdfs clusters as
> storage in the cloud, and i was wondering if any work had been done on
> this.
> Ananth T Sarathy
>