You are viewing a plain text version of this content. The canonical link for it is here.

Posted to hdfs-user@hadoop.apache.org by Jeff Zhang <zj...@gmail.com> on 2010/06/09 11:57:13 UTC

Is there any method to estimate how many files metadata I can store in name node ?

Hi all,

I'd like to estimate how many files I can store in the my name node
with 8G memory, Is there any estimation method ?



-- 
Best Regards

Jeff Zhang

Re: Is there any method to estimate how many files metadata I can store in name node ?

Posted by Jeff Zhang <zj...@gmail.com>.

Thanks Patrick。



On Thu, Jun 10, 2010 at 12:47 AM, Patrick Angeles <pa...@cloudera.com> wrote:
> Hey Jeff,
> A rough (but pretty good) estimation is 1GB per 1M blocks. So if you have
> files that average 1.2GB each and you have a dfs.block.size of 128MB, then
> it would take roughly 10 blocks to store that file, and you would be able to
> store 100k of those files in 1GB of RAM.
> Out of 8GB, I'd assume 7GB is usable for HDFS metadata. The remaining 1GB is
> for operational overhead -- this is probably generous but it's better to
> over-estimate when you're capacity planning.
> Cheers,
> - Patrick
> On Wed, Jun 9, 2010 at 5:57 AM, Jeff Zhang <zj...@gmail.com> wrote:
>>
>> Hi all,
>>
>> I'd like to estimate how many files I can store in the my name node
>> with 8G memory, Is there any estimation method ?
>>
>>
>>
>> --
>> Best Regards
>>
>> Jeff Zhang
>
>



-- 
Best Regards

Jeff Zhang

Re: Is there any method to estimate how many files metadata I can store in name node ?

Posted by Patrick Angeles <pa...@cloudera.com>.

Hey Jeff,

A rough (but pretty good) estimation is 1GB per 1M blocks. So if you have
files that average 1.2GB each and you have a dfs.block.size of 128MB, then
it would take roughly 10 blocks to store that file, and you would be able to
store 100k of those files in 1GB of RAM.

Out of 8GB, I'd assume 7GB is usable for HDFS metadata. The remaining 1GB is
for operational overhead -- this is probably generous but it's better to
over-estimate when you're capacity planning.

Cheers,

- Patrick

On Wed, Jun 9, 2010 at 5:57 AM, Jeff Zhang <zj...@gmail.com> wrote:

> Hi all,
>
> I'd like to estimate how many files I can store in the my name node
> with 8G memory, Is there any estimation method ?
>
>
>
> --
> Best Regards
>
> Jeff Zhang
>