Limiting the total number of tasks per task tracker

[Oliver Senn <se...@student.ethz.ch>]

Hi all,

I'm wondering whether it's possible to limit the number of task that 
are executed in parallel on a task tracker?

I'm using the parameters mapred.tasktracker.{map|reduce}.tasks.maximum
to limit the number of maps and reduces per task tracker but still 
one map and one reduce are scheduled on the same task tracker.

Thanks.

Oliver

Re: Limiting the total number of tasks per task tracker

[Harish Mallipeddi <ha...@gmail.com>]

Setting slowstart to 1 would only control when reduces get started wrt that
particular job's maps (doesn't prevent the reduce task from some other job
to be scheduled alongside the map task from this job on a TT). So this
doesn't meet the OP's requirement - a TT should only run 1 map or 1 reduce
task at a time. There's no easy way to achieve this I'm afraid.

On Sat, Sep 26, 2009 at 3:02 AM, Chandraprakash Bhagtani <
cpbhagtani@gmail.com> wrote:

> you can control that by property
>
> <property>
>
> <name>mapred.reduce.slowstart.completed.maps</name>
>
>  <value>0.05</value>
>
>  <description>Fraction of the number of maps in the job which should be
>
>  complete before reduces are scheduled for the job.
>
>  </description>
>
> </property>
> set the value of this property to 1. It will start the reducers when all
> the
> maps will finish.
> This will work with hadoop version 0.20.0.
>

On Fri, Sep 25, 2009 at 8:39 PM, Steve Loughran <st...@apache.org> wrote:
>
> > Oliver Senn wrote:
> >
> >> Hi,
> >>
> >> Thanks for your answer.
> >>
> >> I used these parameters. But they seem to limit only the number of
> >> parallel maps and parallel reduces separately. They do not prevent the
> >> scheduler from schedule one map and one reduce on the same task tracker
> in
> >> parallel.
> >>
> >> But that's the problem I'm trying to solve. Having at most one task
> >> running on a task tracker at any time (never one map and one reduce
> together
> >> on one task tracker).
> >>
> >>
> > You could do your own scheduler, if you really can't handle the things in
> > parallel
> >
>
>
>
> --
> Thanks & Regards,
> Chandra Prakash Bhagtani,
>



-- 
Harish Mallipeddi
http://blog.poundbang.in

Re: Limiting the total number of tasks per task tracker

[Chandraprakash Bhagtani <cp...@gmail.com>]

you can control that by property

<property>

<name>mapred.reduce.slowstart.completed.maps</name>

  <value>0.05</value>

  <description>Fraction of the number of maps in the job which should be

  complete before reduces are scheduled for the job.

  </description>

</property>
set the value of this property to 1. It will start the reducers when all the
maps will finish.
This will work with hadoop version 0.20.0.

On Fri, Sep 25, 2009 at 8:39 PM, Steve Loughran <st...@apache.org> wrote:

> Oliver Senn wrote:
>
>> Hi,
>>
>> Thanks for your answer.
>>
>> I used these parameters. But they seem to limit only the number of
>> parallel maps and parallel reduces separately. They do not prevent the
>> scheduler from schedule one map and one reduce on the same task tracker in
>> parallel.
>>
>> But that's the problem I'm trying to solve. Having at most one task
>> running on a task tracker at any time (never one map and one reduce together
>> on one task tracker).
>>
>>
> You could do your own scheduler, if you really can't handle the things in
> parallel
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,

Re: Limiting the total number of tasks per task tracker

[Steve Loughran <st...@apache.org>]

Oliver Senn wrote:
> Hi,
> 
> Thanks for your answer.
> 
> I used these parameters. But they seem to limit only the number of 
> parallel maps and parallel reduces separately. They do not prevent the 
> scheduler from schedule one map and one reduce on the same task tracker 
> in parallel.
> 
> But that's the problem I'm trying to solve. Having at most one task 
> running on a task tracker at any time (never one map and one reduce 
> together on one task tracker).
> 

You could do your own scheduler, if you really can't handle the things 
in parallel

Re: Limiting the total number of tasks per task tracker

[Oliver Senn <se...@student.ethz.ch>]

Hi,

Thanks for your answer.

I used these parameters. But they seem to limit only the number of 
parallel maps and parallel reduces separately. They do not prevent 
the scheduler from schedule one map and one reduce on the same task 
tracker in parallel.

But that's the problem I'm trying to solve. Having at most one task 
running on a task tracker at any time (never one map and one reduce 
together on one task tracker).

Any idea how to do this?

Oliver

Chandraprakash Bhagtani wrote:
> Hi Oliver,
> 
> yes you can control number of tasks per tasktracker by
> mapred.tasktracker.{map|reduce}.tasks.maximum.
> in your case
> 
> 1. you might not have restarted the cluster after changing hadoop conf xml.
> 2. your input data might be very less (only one data block might be there)
> therefore only one map task is launched.
> 

Re: Limiting the total number of tasks per task tracker

[Chandraprakash Bhagtani <cp...@gmail.com>]

Hi Oliver,

yes you can control number of tasks per tasktracker by
mapred.tasktracker.{map|reduce}.tasks.maximum.
in your case

1. you might not have restarted the cluster after changing hadoop conf xml.
2. your input data might be very less (only one data block might be there)
therefore only one map task is launched.


On Fri, Sep 25, 2009 at 3:18 PM, Oliver Senn <se...@student.ethz.ch> wrote:

> Hi all,
>
> I'm wondering whether it's possible to limit the number of task that are
> executed in parallel on a task tracker?
>
> I'm using the parameters mapred.tasktracker.{map|reduce}.tasks.maximum
> to limit the number of maps and reduces per task tracker but still one map
> and one reduce are scheduled on the same task tracker.
>
> Thanks.
>
> Oliver
>



-- 
Thanks & Regards,
Chandra Prakash Bhagtani,