You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Arijit Mukherjee <ar...@connectivasystems.com> on 2008/11/04 11:16:45 UTC
Hadoop hardware specs
Hi All
We're thinking of setting up a Hadoop cluster which will be used to
create a prototype system for analyzing telecom data. The wiki page on
machine scaling (http://wiki.apache.org/hadoop/MachineScaling) gives an
overview of the node specs and from the Hadoop primer I found the
following specs -
* 5 x dual core CPUs
* RAM - 4-8GB; ECC preferred, though more expensive
* 2 x 250GB SATA drives (on each of the 5 nodes)
* 1-5 TB external storage
I'm curious to find out what sort of specs do people use normally. Is
the external storage essential or will the individual disks on each node
be sufficient? Why would you need an external storage in a hadoop
cluster? How can I find out what other projects on hadoop are using?
Cheers
Arijit
Dr. Arijit Mukherjee
Principal Member of Technical Staff, Level-II
Connectiva Systems (I) Pvt. Ltd.
J-2, Block GP, Sector V, Salt Lake
Kolkata 700 091, India
Phone: +91 (0)33 23577531/32 x 107
http://www.connectivasystems.com
Re: Hadoop hardware specs
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Roger,
SSH is only needed to start and stop daemons - it's not really needed
for running Hadoop itself. Currently, we do this through custom site
mechanisms, and not through SSH.
Brian
On Nov 4, 2008, at 10:36 AM, Zhang, Roger wrote:
> Brian,
>
> You seem to have a pretty large cluster. How do you think about the
> overall performance?
> Is your implementation on Open-SSH or SSH2?
>
> I'm new to this and trying to setup a 20 node cluster. But our Linux
> boxes enforced F-secure SSH2 already, which I found HDFS 0.18 does
> not support right now.
>
> Anyone has any idea of a workaround?
>
>
> Thanks and best Rgds.
> Roger Zhang
>
> -----Original Message-----
> From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
> Sent: 2008年11月4日 21:36
> To: core-user@hadoop.apache.org
> Subject: Re: Hadoop hardware specs
>
> Hey Arjit,
>
> We use all internal SATA drives in our cluster, which is about 110TB
> today; if we grow it to our planned 350TB, it will be a healthy mix of
> worker nodes w/ SATA, large internal chases (12 - 48TB), SCSI attached
> vaults, and fibre channel vaults.
>
> Brian
>
> On Nov 4, 2008, at 4:16 AM, Arijit Mukherjee wrote:
>
>> Hi All
>>
>> We're thinking of setting up a Hadoop cluster which will be used to
>> create a prototype system for analyzing telecom data. The wiki page
>> on
>> machine scaling (http://wiki.apache.org/hadoop/MachineScaling) gives
>> an
>> overview of the node specs and from the Hadoop primer I found the
>> following specs -
>>
>> * 5 x dual core CPUs
>> * RAM - 4-8GB; ECC preferred, though more expensive
>> * 2 x 250GB SATA drives (on each of the 5 nodes)
>> * 1-5 TB external storage
>>
>> I'm curious to find out what sort of specs do people use normally. Is
>> the external storage essential or will the individual disks on each
>> node
>> be sufficient? Why would you need an external storage in a hadoop
>> cluster? How can I find out what other projects on hadoop are using?
>> Cheers
>> Arijit
>>
>>
>> Dr. Arijit Mukherjee
>> Principal Member of Technical Staff, Level-II
>> Connectiva Systems (I) Pvt. Ltd.
>> J-2, Block GP, Sector V, Salt Lake
>> Kolkata 700 091, India
>> Phone: +91 (0)33 23577531/32 x 107
>> http://www.connectivasystems.com
>>
>
RE: Hadoop hardware specs
Posted by "Zhang, Roger" <ro...@intel.com>.
Brian,
You seem to have a pretty large cluster. How do you think about the overall performance?
Is your implementation on Open-SSH or SSH2?
I'm new to this and trying to setup a 20 node cluster. But our Linux boxes enforced F-secure SSH2 already, which I found HDFS 0.18 does not support right now.
Anyone has any idea of a workaround?
Thanks and best Rgds.
Roger Zhang
-----Original Message-----
From: Brian Bockelman [mailto:bbockelm@cse.unl.edu]
Sent: 2008年11月4日 21:36
To: core-user@hadoop.apache.org
Subject: Re: Hadoop hardware specs
Hey Arjit,
We use all internal SATA drives in our cluster, which is about 110TB
today; if we grow it to our planned 350TB, it will be a healthy mix of
worker nodes w/ SATA, large internal chases (12 - 48TB), SCSI attached
vaults, and fibre channel vaults.
Brian
On Nov 4, 2008, at 4:16 AM, Arijit Mukherjee wrote:
> Hi All
>
> We're thinking of setting up a Hadoop cluster which will be used to
> create a prototype system for analyzing telecom data. The wiki page on
> machine scaling (http://wiki.apache.org/hadoop/MachineScaling) gives
> an
> overview of the node specs and from the Hadoop primer I found the
> following specs -
>
> * 5 x dual core CPUs
> * RAM - 4-8GB; ECC preferred, though more expensive
> * 2 x 250GB SATA drives (on each of the 5 nodes)
> * 1-5 TB external storage
>
> I'm curious to find out what sort of specs do people use normally. Is
> the external storage essential or will the individual disks on each
> node
> be sufficient? Why would you need an external storage in a hadoop
> cluster? How can I find out what other projects on hadoop are using?
> Cheers
> Arijit
>
>
> Dr. Arijit Mukherjee
> Principal Member of Technical Staff, Level-II
> Connectiva Systems (I) Pvt. Ltd.
> J-2, Block GP, Sector V, Salt Lake
> Kolkata 700 091, India
> Phone: +91 (0)33 23577531/32 x 107
> http://www.connectivasystems.com
>
Re: Hadoop hardware specs
Posted by Brian Bockelman <bb...@cse.unl.edu>.
Hey Arjit,
We use all internal SATA drives in our cluster, which is about 110TB
today; if we grow it to our planned 350TB, it will be a healthy mix of
worker nodes w/ SATA, large internal chases (12 - 48TB), SCSI attached
vaults, and fibre channel vaults.
Brian
On Nov 4, 2008, at 4:16 AM, Arijit Mukherjee wrote:
> Hi All
>
> We're thinking of setting up a Hadoop cluster which will be used to
> create a prototype system for analyzing telecom data. The wiki page on
> machine scaling (http://wiki.apache.org/hadoop/MachineScaling) gives
> an
> overview of the node specs and from the Hadoop primer I found the
> following specs -
>
> * 5 x dual core CPUs
> * RAM - 4-8GB; ECC preferred, though more expensive
> * 2 x 250GB SATA drives (on each of the 5 nodes)
> * 1-5 TB external storage
>
> I'm curious to find out what sort of specs do people use normally. Is
> the external storage essential or will the individual disks on each
> node
> be sufficient? Why would you need an external storage in a hadoop
> cluster? How can I find out what other projects on hadoop are using?
> Cheers
> Arijit
>
>
> Dr. Arijit Mukherjee
> Principal Member of Technical Staff, Level-II
> Connectiva Systems (I) Pvt. Ltd.
> J-2, Block GP, Sector V, Salt Lake
> Kolkata 700 091, India
> Phone: +91 (0)33 23577531/32 x 107
> http://www.connectivasystems.com
>
Re: Hadoop hardware specs
Posted by Allen Wittenauer <aw...@yahoo-inc.com>.
On 11/4/08 2:16 AM, "Arijit Mukherjee" <ar...@connectivasystems.com>
wrote:
> * 1-5 TB external storage
>
> I'm curious to find out what sort of specs do people use normally. Is
> the external storage essential or will the individual disks on each node
> be sufficient? Why would you need an external storage in a hadoop
> cluster?
The big reason for the external storage is two fold:
A) Provide shared home directory (especially for the HDFS user so that it is
easy to use the start scripts that call ssh)
B) An off-machine copy of the fsimage and edits file as used by the name
node. This way if the name node goes belly up, you'll have an always
up-to-date backup to recover.
> How can I find out what other projects on hadoop are using?
Slide 12 of the Apachecon presentation I did earlier this year talks
about what Yahoo!'s typical node looks like. For a small 5 node cluster,
your hardware specs seem fine to me.
An 8GB namenode for 4 data nodes (or maybe even running nn on the same
machine as a data node if memory size of jobs is kept in check) should be
a-ok, even if you double the storage. You're likely going to run out of
disk space before the name node starts swapping.
RE: Hadoop hardware specs
Posted by Arijit Mukherjee <ar...@connectivasystems.com>.
One correction - the number 5 in the mail below is my estimation of the
number of nodes we might need. Can this be too small a cluster?
Arijit
Dr. Arijit Mukherjee
Principal Member of Technical Staff, Level-II
Connectiva Systems (I) Pvt. Ltd.
J-2, Block GP, Sector V, Salt Lake
Kolkata 700 091, India
Phone: +91 (0)33 23577531/32 x 107
http://www.connectivasystems.com
-----Original Message-----
From: Arijit Mukherjee [mailto:armukherjee@connectivasystems.com]
Sent: Tuesday, November 04, 2008 3:47 PM
To: core-user@hadoop.apache.org
Subject: Hadoop hardware specs
Hi All
We're thinking of setting up a Hadoop cluster which will be used to
create a prototype system for analyzing telecom data. The wiki page on
machine scaling (http://wiki.apache.org/hadoop/MachineScaling) gives an
overview of the node specs and from the Hadoop primer I found the
following specs -
* 5 x dual core CPUs
* RAM - 4-8GB; ECC preferred, though more expensive
* 2 x 250GB SATA drives (on each of the 5 nodes)
* 1-5 TB external storage
I'm curious to find out what sort of specs do people use normally. Is
the external storage essential or will the individual disks on each node
be sufficient? Why would you need an external storage in a hadoop
cluster? How can I find out what other projects on hadoop are using?
Cheers
Arijit
Dr. Arijit Mukherjee
Principal Member of Technical Staff, Level-II
Connectiva Systems (I) Pvt. Ltd.
J-2, Block GP, Sector V, Salt Lake
Kolkata 700 091, India
Phone: +91 (0)33 23577531/32 x 107 http://www.connectivasystems.com No
virus found in this incoming message. Checked by AVG -
http://www.avg.com
Version: 8.0.175 / Virus Database: 270.8.6/1765 - Release Date:
11/3/2008 4:59 PM