You are viewing a plain text version of this content. The canonical link for it is here.

Posted to common-user@hadoop.apache.org by Manuel Simoni <ma...@systemone.at> on 2006/05/16 19:38:56 UTC

Is HDFS suited to large numbers of small files with random access?

Hi,

I am looking for a distributed file system for large numbers of small 
files (a few KB each). Maybe the files would be rows in HDFS.

But my access patterns are different from MapRed-style jobs: reading and 
writing files rather randomly instead of sequential processing.

Is HDFS suited to this kind of workload?

Thanks,
Manuel

Re: Is HDFS suited to large numbers of small files with random access?

Posted by Richard Jones <rj...@last.fm>.

You might like to try MogileFS

http://www.danga.com/mogilefs/

On Tuesday 16 May 2006 18:38, Manuel Simoni wrote:
> Hi,
>
> I am looking for a distributed file system for large numbers of small
> files (a few KB each). Maybe the files would be rows in HDFS.
>
> But my access patterns are different from MapRed-style jobs: reading and
> writing files rather randomly instead of sequential processing.
>
> Is HDFS suited to this kind of workload?
>
> Thanks,
> Manuel

-- 
Richard Jones
Last.fm Ltd. | http://www.last.fm/
Office: +44 (0) 207 7780 7080
Mobile: +44 (0) 7740 194 389