You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Robert Krüger <kr...@signal7.de> on 2008/03/27 19:40:57 UTC
Do multiple small files share one block?
Hi,
this seems like an FAQ but I didn't explicitly see it in the docs: Is
the minmium size a file occupies on HDFS controlled by the block size,
i.e. would using the default block size of 64 B result in consumption of
64 MB if I stored a file of 1 byte? I would assume yes based on the fact
that HDFS was primarily developed for the distribution of large files
and sentences like "Internally, a file is split into one or more blocks
and these blocks are stored in a set of Datanodes" seem to imply it but
I can't find a definite answer. If so and the default block size is 64MB
it feels like abusing HDFS if we set the block size to 10k just because
a few hundred thousand files of ours are only a few K in size but I
should probably run some benchmarks.
Thanks in advance,
Robert
Re: Do multiple small files share one block?
Posted by Doug Cutting <cu...@apache.org>.
Robert Krüger wrote:
> this seems like an FAQ but I didn't explicitly see it in the docs: Is
> the minmium size a file occupies on HDFS controlled by the block size,
> i.e. would using the default block size of 64 B result in consumption of
> 64 MB if I stored a file of 1 byte?
No. The last block in a file is only be as long as it needs to be.
Doug