multi core nodes

[ll_oz_ll <hi...@yahoo.com>]

Hi,
Is hadoop able to take into account multi core nodes, so that nodes which
have multiple cores run multiple concurrent jobs ?
Or does that need to be configured manually and if so can that be configured
individually for each node ?

thanks. I would really appreciate any clarification on this.


-- 
View this message in context: http://www.nabble.com/multi-core-nodes-tp25283387p25283387.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.

RE: multi core nodes

[Amogh Vasekar <am...@yahoo-inc.com>]

Before setting the task limits, do take into account the memory considerations ( many archive posts on this can be found ). 
Also, your tasktracker and datanode daemons will run on that machine as well, so you might want to set aside some processing power for that.

Cheers!
Amogh

-----Original Message-----
From: Erik Forsberg [mailto:forsberg@opera.com] 
Sent: Friday, September 04, 2009 11:55 AM
To: common-user@hadoop.apache.org
Subject: Re: multi core nodes

On Thu, 3 Sep 2009 13:20:16 -0700 (PDT)
ll_oz_ll <hi...@yahoo.com> wrote:

> 
> Hi,
> Is hadoop able to take into account multi core nodes, so that nodes
> which have multiple cores run multiple concurrent jobs ?
> Or does that need to be configured manually and if so can that be
> configured individually for each node ?

Yes, it has to be configured manually. You set the following two
configuration variables in hadoop-site.xml on each node depending on the
number of cores on the node:

mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum

According to the book "Hadoop - the definitive guide", a good rule of
thumb is to have between 1 and 2 tasks per processor, counting both map
and reduce tasks. So, for example, if a machine has 8 cores, setting
mapred.tasktracker.map.tasks.maximum = 8 and
mapred.tasktracker.reduce.tasks.maximum = 8 probably makes sense, but
this also depends a bit on your load.

Cheers,
\EF
-- 
Erik Forsberg <fo...@opera.com>
Developer, Opera Software - http://www.opera.com/

Re: multi core nodes

[Erik Forsberg <fo...@opera.com>]

On Thu, 3 Sep 2009 13:20:16 -0700 (PDT)
ll_oz_ll <hi...@yahoo.com> wrote:

> 
> Hi,
> Is hadoop able to take into account multi core nodes, so that nodes
> which have multiple cores run multiple concurrent jobs ?
> Or does that need to be configured manually and if so can that be
> configured individually for each node ?

Yes, it has to be configured manually. You set the following two
configuration variables in hadoop-site.xml on each node depending on the
number of cores on the node:

mapred.tasktracker.map.tasks.maximum
mapred.tasktracker.reduce.tasks.maximum

According to the book "Hadoop - the definitive guide", a good rule of
thumb is to have between 1 and 2 tasks per processor, counting both map
and reduce tasks. So, for example, if a machine has 8 cores, setting
mapred.tasktracker.map.tasks.maximum = 8 and
mapred.tasktracker.reduce.tasks.maximum = 8 probably makes sense, but
this also depends a bit on your load.

Cheers,
\EF
-- 
Erik Forsberg <fo...@opera.com>
Developer, Opera Software - http://www.opera.com/

Re: multi core nodes

[Starry SHI <st...@gmail.com>]

I also would like to know whether it is possible to configure this. Hope
somebody can provide a solution.

Starry


On Fri, Sep 4, 2009 at 04:20, ll_oz_ll <hi...@yahoo.com> wrote:

>
> Hi,
> Is hadoop able to take into account multi core nodes, so that nodes which
> have multiple cores run multiple concurrent jobs ?
> Or does that need to be configured manually and if so can that be
> configured
> individually for each node ?
>
> thanks. I would really appreciate any clarification on this.
>
>
> --
> View this message in context:
> http://www.nabble.com/multi-core-nodes-tp25283387p25283387.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>