You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Steve Schlosser <sw...@gmail.com> on 2008/02/07 16:23:52 UTC

Override mapred.tasktracker.tasks.maximum?

Hello all

Is it possible for a Hadoop program to override
mapred.tasktracker.tasks.maximum at runtime?  I've found that my job
overloads our nodes when running our default 8 tasks per node, but if
I decrease mapred.tasktracker.tasks.maximum in the config file and
restart the cluster it seems to go fine.  However, when I tried to set
this parameter in my program it seemed to be ignored.  I'd like to set
this at runtime rather than having to restart the cluster.  Is this
only read when tasktrackers are started or can I override it?

Thanks!

-steve

Re: Override mapred.tasktracker.tasks.maximum?

Posted by Jason Venner <ja...@attributor.com>.
This should be one of the features coming in 0.16 via HOD

Steve Schlosser wrote:
> Hello all
>
> Is it possible for a Hadoop program to override
> mapred.tasktracker.tasks.maximum at runtime?  I've found that my job
> overloads our nodes when running our default 8 tasks per node, but if
> I decrease mapred.tasktracker.tasks.maximum in the config file and
> restart the cluster it seems to go fine.  However, when I tried to set
> this parameter in my program it seemed to be ignored.  I'd like to set
> this at runtime rather than having to restart the cluster.  Is this
> only read when tasktrackers are started or can I override it?
>
> Thanks!
>
> -steve
>

Re: Override mapred.tasktracker.tasks.maximum?

Posted by Arun C Murthy <ar...@yahoo-inc.com>.
On Feb 7, 2008, at 7:23 AM, Steve Schlosser wrote:

> Hello all
>
> Is it possible for a Hadoop program to override
> mapred.tasktracker.tasks.maximum at runtime?  I've found that my job
> overloads our nodes when running our default 8 tasks per node, but if
> I decrease mapred.tasktracker.tasks.maximum in the config file and
> restart the cluster it seems to go fine.  However, when I tried to set
> this parameter in my program it seemed to be ignored.

No, mapred.tasktracker.tasks.maximum (mapred.tasktracker.{map|reduce} 
tasks.maximum in 0.16.0) is a property of the cluster and not a job- 
specific property. Hence you cannot override it.

As Jason mentioned, you could use Haddop-On-Demand (a resource  
utilization framework coming in 0.16.0) to launch a new cluster per  
job and then configure each job (and hence cluster) differently...

Arun

> I'd like to set
> this at runtime rather than having to restart the cluster.  Is this
> only read when tasktrackers are started or can I override it?
>
> Thanks!
>
> -steve