You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Nitin Khandelwal <ni...@germinait.com> on 2011/12/20 12:46:57 UTC
One task per Tasktracker
Hey,
We use capacity scheduler and divide our map slots among queues. For a
particular kind of job, we want to schedule at most one task per task
tracker. How does one do this? We are using Hadoop 0.20.205.0.
Thanks,
--
Nitin Khandelwal
Re: One task per Tasktracker
Posted by Nitin Khandelwal <ni...@germinait.com>.
Hi Harsh,
Thanks for your reply.
The task makes web request which is IP limited. So , I need only one task
per slave.
Thanks,
Nitin
On 20 December 2011 17:37, Harsh J <ha...@cloudera.com> wrote:
> Nitin,
>
> AFAIK there's no way to do this directly. Why are you looking to do
> that though? What gains do you see you'll get that way against letting
> slots be used in the regular way, given multi-core CPUs and multi-disk
> DFS setups?
>
> On Tue, Dec 20, 2011 at 5:16 PM, Nitin Khandelwal
> <ni...@germinait.com> wrote:
> > Hey,
> >
> > We use capacity scheduler and divide our map slots among queues. For a
> > particular kind of job, we want to schedule at most one task per task
> > tracker. How does one do this? We are using Hadoop 0.20.205.0.
> >
> > Thanks,
> >
> > --
> > Nitin Khandelwal
> >
> >
>
>
>
> --
> Harsh J
>
--
Nitin Khandelwal
Re: One task per Tasktracker
Posted by Harsh J <ha...@cloudera.com>.
Nitin,
AFAIK there's no way to do this directly. Why are you looking to do
that though? What gains do you see you'll get that way against letting
slots be used in the regular way, given multi-core CPUs and multi-disk
DFS setups?
On Tue, Dec 20, 2011 at 5:16 PM, Nitin Khandelwal
<ni...@germinait.com> wrote:
> Hey,
>
> We use capacity scheduler and divide our map slots among queues. For a
> particular kind of job, we want to schedule at most one task per task
> tracker. How does one do this? We are using Hadoop 0.20.205.0.
>
> Thanks,
>
> --
> Nitin Khandelwal
>
>
--
Harsh J
Re: One task per Tasktracker
Posted by Nitin Khandelwal <ni...@germinait.com>.
Hi Arun,
I didnot get you completely. Are you trying to say that i should assign
memory limits to map task so , that more than one wont be posted on each TT
? Also, I have multiple type of jobs (each in separate queue) which are
IP limited. I want one map task for each type of job in a TT. So, if i
assign memory limit to each type of job, will it work ? What i want to ask
is that if one type of job is not running, is it possible at scheduler will
assign 2 slots on same machine to some other type of job ?
Thanks,
Nitin
On 20 December 2011 21:43, Arun C Murthy <ac...@hortonworks.com> wrote:
> Just use multiple slots per each map.
>
> See:
> http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html#Resource+based+scheduling
>
> Arun
>
> On Dec 20, 2011, at 3:46 AM, Nitin Khandelwal wrote:
>
> > Hey,
> >
> > We use capacity scheduler and divide our map slots among queues. For a
> particular kind of job, we want to schedule at most one task per task
> tracker. How does one do this? We are using Hadoop 0.20.205.0.
> >
> > Thanks,
> >
> > --
> > Nitin Khandelwal
> >
> >
>
>
--
Nitin Khandelwal
Re: One task per Tasktracker
Posted by Arun C Murthy <ac...@hortonworks.com>.
Just use multiple slots per each map.
See: http://hadoop.apache.org/common/docs/stable/capacity_scheduler.html#Resource+based+scheduling
Arun
On Dec 20, 2011, at 3:46 AM, Nitin Khandelwal wrote:
> Hey,
>
> We use capacity scheduler and divide our map slots among queues. For a particular kind of job, we want to schedule at most one task per task tracker. How does one do this? We are using Hadoop 0.20.205.0.
>
> Thanks,
>
> --
> Nitin Khandelwal
>
>