commodity vs. high perf machines: which would you rather

[Chris Fellows <ch...@yahoo.com>]

Hello,

Much of the hadoop documentation speaks to large clusters of commodity machines. There is a debate on our end about which would be better: a small number of high performance machines (2 boxes with 4 quad core processors) or X number of commodity machines. I feel that disk I/O might be the bottle neck with the 2 high perf machines (though I did just read in the FAQ about being able to split the dfs-data across multiple drives).

So this is a "which would rather" question. If you were setting up a cluster of machines to perform data rollups/aggregation (and other mapred tasks) on files in the .25-1TB size, which would rather have:

1. 2 4 quad core machines with your choice on RAM and number of drives
2. 10 (or more) commodity machines (as defined on the hadoop wiki)

And of course a "why?" would be very helpful.

Thanks!

Re: commodity vs. high perf machines: which would you rather

[Ted Dunning <td...@veoh.com>]

I find that most of my hadoop applications are VERY small in memory
consumption.  I have almost all 4-8 GB machines, but 2GB would have been
over-kill.


On 11/7/07 12:39 PM, "Jonathan Hendler" <he...@yahoo.com> wrote:

> I see from other posts ECC is important, and in general RAM can't hurt.
> An SSD would be nice too, to help with disk i/o.

Re: commodity vs. high perf machines: which would you rather

[Jonathan Hendler <he...@yahoo.com>]

I like these high level questions, as I have yet do do an actual
install, but being an utter newbie, maybe my perspective will contribute
something.

I'd say option 3, which is:
1. One failsafe, strong machine for a master.
2. As many "commodity machines" as you can muster. With an emphasis on
RAM and disk IO.

My justification is that since the architecture is not P2P and requires
Master to do all the scheduling. Of course, I could not understand
something, and master might actually work fine as a weaker machine,
which only does scheduling and routing, while the workers need to be
bulky. I don't know the answer, but I suspect that kind of thinking
might point you in the right direction.

Another of other factors also get left out of decisions like this:
1. long term planning - how much would you scale in the future? Is this
just a demo, or the production environment?
2. network environment - what kind of network will the machines be
living in?
3. What is the MapReduce algorithm you are doing? Normally disk i/o
bound, but maybe you're doing something "difficult" for CPU.

I see from other posts ECC is important, and in general RAM can't hurt.
An SSD would be nice too, to help with disk i/o.

HTH

- Jonathan

Chris Fellows wrote:
> Hello,
>
> Much of the hadoop documentation speaks to large clusters of commodity machines. There is a debate on our end about which would be better: a small number of high performance machines (2 boxes with 4 quad core processors) or X number of commodity machines. I feel that disk I/O might be the bottle neck with the 2 high perf machines (though I did just read in the FAQ about being able to split the dfs-data across multiple drives).
>
> So this is a "which would rather" question. If you were setting up a cluster of machines to perform data rollups/aggregation (and other mapred tasks) on files in the .25-1TB size, which would rather have:
>
> 1. 2 4 quad core machines with your choice on RAM and number of drives
> 2. 10 (or more) commodity machines (as defined on the hadoop wiki)
>
> And of course a "why?" would be very helpful.
>
> Thanks!
>
>
>   

Re: commodity vs. high perf machines: which would you rather

[Ted Dunning <td...@veoh.com>]

My mileage computation came out essentially the same as what Doug says.

I use some cast-off machines that were not reliable enough for other
applications.  They originally cost us about 2/3 what our normal production
boxes cots and achieve almost exactly 1/2  as much.  Our production boxes
are typically dual CPU's with dual cores.


On 11/7/07 12:27 PM, "Doug Cutting" <cu...@apache.org> wrote:

>   So I'd hazard that moderately high-end commodity hardware is the most
> cost-effective for Hadoop today.  YMMV.

Re: commodity vs. high perf machines: which would you rather

[Doug Cutting <cu...@apache.org>]

Chris Fellows wrote:
> 1. 2 4 quad core machines with your choice on RAM and number of drives
> 2. 10 (or more) commodity machines (as defined on the hadoop wiki)

I'm not sure what definition you're referring to.  Quad-core processors 
are a commodity today, aren't they?  So I think your question is perhaps 
where on the spectrum of commodity hardware is it preferable to be, on 
the high- or low-end?

Hadoop benefits greatly from ECC memory, which is not low-end. 
Multi-core boxes tend to give you more computation per dollar, per watt 
and per unit of operational maintenance.  But the highest clockrate 
processors tend to not be cost-effective, as do the very largest drives. 
  So I'd hazard that moderately high-end commodity hardware is the most 
cost-effective for Hadoop today.  YMMV.

Doug

Re: commodity vs. high perf machines: which would you rather

[Raghu Angadi <ra...@yahoo-inc.com>]

Does commodity hardware come with ECC memory? Since Hadoop apps tend to 
move large amounts of data around, ECC memory seems pretty important.

With just two machines, you might be limited since you need to run 
multiple components (Namenode, job tracker, etc) on one machine.
Two machines seems quite low... I would go with 10, if cost is same. 
Also 10 machines might eat up more power.

Raghu.

Chris Fellows wrote:
> Hello,
> 
> Much of the hadoop documentation speaks to large clusters of commodity machines. There is a debate on our end about which would be better: a small number of high performance machines (2 boxes with 4 quad core processors) or X number of commodity machines. I feel that disk I/O might be the bottle neck with the 2 high perf machines (though I did just read in the FAQ about being able to split the dfs-data across multiple drives).
> 
> So this is a "which would rather" question. If you were setting up a cluster of machines to perform data rollups/aggregation (and other mapred tasks) on files in the .25-1TB size, which would rather have:
> 
> 1. 2 4 quad core machines with your choice on RAM and number of drives
> 2. 10 (or more) commodity machines (as defined on the hadoop wiki)
> 
> And of course a "why?" would be very helpful.
> 
> Thanks!
> 
> 

Re: commodity vs. high perf machines: which would you rather

[Ted Dunning <td...@veoh.com>]

For me, I have three configurations available.

A) database class machine with many (>10) fast SAS drives and >10GB memory,
dual or quad x quad core cpu's.  Let's say that this costs about 20K$.

B) generic productiion machine with 2 x 250GB SATA drives, 4-12GB RAM, dual
x dual core CPU's (=Dell 1950).  Cost is about 2K$.

C) POS beige box machine with 2 x SATA drives of variable size, 4 GB RAM,
single dual core CPU.  Cost is about 1K$.

For a $50K budget, I would take 25x(b) over 50x(c) due to simpler and
smaller admin issues even though cost/performance would be nominally about
the same.  I would avoid 2x(a) like the plague.





On 11/7/07 11:56 AM, "Chris Fellows" <ch...@yahoo.com> wrote:

> Hello,
> 
> Much of the hadoop documentation speaks to large clusters of commodity
> machines. There is a debate on our end about which would be better: a small
> number of high performance machines (2 boxes with 4 quad core processors) or X
> number of commodity machines. I feel that disk I/O might be the bottle neck
> with the 2 high perf machines (though I did just read in the FAQ about being
> able to split the dfs-data across multiple drives).
> 
> So this is a "which would rather" question. If you were setting up a cluster
> of machines to perform data rollups/aggregation (and other mapred tasks) on
> files in the .25-1TB size, which would rather have:
> 
> 1. 2 4 quad core machines with your choice on RAM and number of drives
> 2. 10 (or more) commodity machines (as defined on the hadoop wiki)
> 
> And of course a "why?" would be very helpful.
> 
> Thanks!
>