Hardware specs

[Mark <st...@gmail.com>]

Can someone give some brief guidance on some (ballpark) specs for a 
hadoop cluster?

- What are the typical size and memory requirements are there for a 
dedicated namenode?
- What are the typical size and memory requirements are there for 
individual datanodes?
- Should there be more, less or even amounts of ram on each datanode?
- Should the namenode have more, less or an even amount of ram as 
compared to the datanode?

Thanks

Re: Hardware specs

[Steve Loughran <st...@apache.org>]

On 06/09/2011 03:59 PM, Mark wrote:
> Can someone give some brief guidance on some (ballpark) specs for a
> hadoop cluster?
>
> - What are the typical size and memory requirements are there for a
> dedicated namenode?

-this depends on cluster size

> - What are the typical size and memory requirements are there for
> individual datanodes?

this depends on the jobs you run, whether they are MR or HBase, and is a 
bit of a moving target as the #of cores and HDDs increases. I'm going to 
avoid making any recommendations.

> - Should there be more, less or even amounts of ram on each datanode?

it's good to have them similar, at least with a stable amount of RAM per 
core.

> - Should the namenode have more, less or an even amount of ram as
> compared to the datanode?

more and you want the storage to be HA/raided, something else reliable 
remote mounted. you need lots of swap space too.

the secondary namenode isn't  failover node, but is needed to bring up 
clusters faster on a restart, it should have similar HW specs to the 
namenode.

there's also the job tracker. In big clusters this can use a lot of RAM, 
depending on the #of jobs and the #of counters each job has.

Re: Hardware specs

[Joey Echeverria <jo...@cloudera.com>]

There are some good recommendations in this blog post:

http://www.cloudera.com/blog/2010/03/clouderas-support-team-shares-some-basic-hardware-recommendations/

It's a little dated, but the reasoning and basics are sound.

-Joey

On Thu, Jun 9, 2011 at 10:59 AM, Mark <st...@gmail.com> wrote:
> Can someone give some brief guidance on some (ballpark) specs for a hadoop
> cluster?
>
> - What are the typical size and memory requirements are there for a
> dedicated namenode?
> - What are the typical size and memory requirements are there for individual
> datanodes?
> - Should there be more, less or even amounts of ram on each datanode?
> - Should the namenode have more, less or an even amount of ram as compared
> to the datanode?
>
> Thanks
>
>



-- 
Joseph Echeverria
Cloudera, Inc.
443.305.9434