Thoughts about Hadoop cluster hardware

[u235sentinel <u2...@gmail.com>]

So we're talking to Dell about their new PowerEdge c2100 servers for a 
Hadoop cluster but I'm wondering.  Isn't this still a little overboard 
for nodes in a cluster?  I'm wondering if we bought say 100 poweredge 
2750's instead of just 50 c2100's.  The price would be about the same 
for the configuration we're talking about and we would get twice as many 
nodes.

I'm curious if any other's are running Dell PowerEdge servers with Hadoop.

We've also been kicking the idea around of going with blade servers 
(Dell and/or HP).

Just curious

Thanks!!

Re: Thoughts about Hadoop cluster hardware

[U235Sentinel <u2...@gmail.com>]

Awesome!  I appreciate it.  I'm off on training right now so I'm just
starting to catch up.  I'll check out those servers and see how they compare

thanks a bunch!

On Tue, Jul 13, 2010 at 8:36 PM, Allen Wittenauer
<aw...@linkedin.com>wrote:

>
> On Jul 13, 2010, at 5:00 PM, u235sentinel wrote:
>
> > So we're talking to Dell about their new PowerEdge c2100 servers for a
> Hadoop cluster but I'm wondering.  Isn't this still a little overboard for
> nodes in a cluster?  I'm wondering if we bought say 100 poweredge 2750's
> instead of just 50 c2100's.  The price would be about the same for the
> configuration we're talking about and we would get twice as many nodes.
>
> Ultimately, it depends upon your job flow and how much data you have.
>
> FWIW we're currently using a Sun equivalent of the C2100s w/8 of the 12
> drive slots filled.  You need a *LOT* of iops to make it worth while.  [From
> what I've seen, even people who think they have a lot of iops generally have
> other problems with their code/tuning that are causing the iops.   So even
> if you think you have a lot, you may not.]
>
> > I'm curious if any other's are running Dell PowerEdge servers with
> Hadoop.
> >
> > We've also been kicking the idea around of going with blade servers (Dell
> and/or HP).
>
> If you are thinking traditional blade where storage is comes mainly from
> NAS or SAN, you are going to be very, very unhappy unless your data set is
> very, very tiny.
>
> Check out the PoweredBy page on the wiki.  Quite a few folks list their
> gear. FWIW, we're currently evaluating HP SLs and should be getting some
> Dell C6100s in soon, assuming Dell can deliver the eval unit on time.

Re: Thoughts about Hadoop cluster hardware

[Allen Wittenauer <aw...@linkedin.com>]

On Jul 13, 2010, at 5:00 PM, u235sentinel wrote:

> So we're talking to Dell about their new PowerEdge c2100 servers for a Hadoop cluster but I'm wondering.  Isn't this still a little overboard for nodes in a cluster?  I'm wondering if we bought say 100 poweredge 2750's instead of just 50 c2100's.  The price would be about the same for the configuration we're talking about and we would get twice as many nodes.

Ultimately, it depends upon your job flow and how much data you have.  

FWIW we're currently using a Sun equivalent of the C2100s w/8 of the 12 drive slots filled.  You need a *LOT* of iops to make it worth while.  [From what I've seen, even people who think they have a lot of iops generally have other problems with their code/tuning that are causing the iops.   So even if you think you have a lot, you may not.]

> I'm curious if any other's are running Dell PowerEdge servers with Hadoop.
> 
> We've also been kicking the idea around of going with blade servers (Dell and/or HP).

If you are thinking traditional blade where storage is comes mainly from NAS or SAN, you are going to be very, very unhappy unless your data set is very, very tiny.

Check out the PoweredBy page on the wiki.  Quite a few folks list their gear. FWIW, we're currently evaluating HP SLs and should be getting some Dell C6100s in soon, assuming Dell can deliver the eval unit on time.