Larger Clusters with Different CPUs

[Xavier Stevens <Xa...@fox.com>]

Does anyone have any suggestions/best practices when configuring sets of
machines with varying number of CPU cores?

Basically I have two types of machines.
1) 8-cores
2) 2-cores

And I would like to make sure that the number of tasks for the 8-cores
is 8 and for 2-cores is 2. 

How are others handling this type of situation?

-Xavier

RE: Larger Clusters with Different CPUs

[Xavier Stevens <Xa...@fox.com>]

How exactly do you do the per node configuration?  

Currently each machine in my cluster has an NFS mount for HADOOP_HOME so
all of the machines use the same configuration.  I am assuming I would
need to make a particular config file like hadoop-site.xml local to each
machine.  Unless there is a way to specify per machine in a single
config file.

-Xavier


-----Original Message-----
From: Ted Dunning 
Sent: Wednesday, January 23, 2008 12:11 PM
To: core-user@hadoop.apache.org
Subject: Re: Larger Clusters with Different CPUs


I don't know when it became effective, but you can configure number of
tasks per node.

I would recommend slight overloads on your boxes, btw.  Something like
9-10 and 3 tasks for the two kinds of boxes.  That gives the linux
scheduler a little bit of stuff to fill in the cracks with.  This
matters most if your maps are very short ... as one exits, it is nice to
have a replacement already running.


On 1/23/08 12:02 PM, "Xavier Stevens" <Xa...@fox.com> wrote:

> Does anyone have any suggestions/best practices when configuring sets 
> of machines with varying number of CPU cores?
> 
> Basically I have two types of machines.
> 1) 8-cores
> 2) 2-cores
> 
> And I would like to make sure that the number of tasks for the 8-cores

> is 8 and for 2-cores is 2.
> 
> How are others handling this type of situation?
> 
> -Xavier
> 

Re: Larger Clusters with Different CPUs

[Ted Dunning <td...@veoh.com>]

I don't know when it became effective, but you can configure number of tasks
per node.

I would recommend slight overloads on your boxes, btw.  Something like 9-10
and 3 tasks for the two kinds of boxes.  That gives the linux scheduler a
little bit of stuff to fill in the cracks with.  This matters most if your
maps are very short ... as one exits, it is nice to have a replacement
already running.


On 1/23/08 12:02 PM, "Xavier Stevens" <Xa...@fox.com> wrote:

> Does anyone have any suggestions/best practices when configuring sets of
> machines with varying number of CPU cores?
> 
> Basically I have two types of machines.
> 1) 8-cores
> 2) 2-cores
> 
> And I would like to make sure that the number of tasks for the 8-cores
> is 8 and for 2-cores is 2.
> 
> How are others handling this type of situation?
> 
> -Xavier
>