You are viewing a plain text version of this content. The canonical link for it is here.
Posted to mapreduce-user@hadoop.apache.org by Steve Lewis <lo...@gmail.com> on 2011/11/10 04:06:27 UTC
Any way to limit the total tasks running on a node in 0.202
Hadoop can set the maximum mappers and reducers running on a node but under
0.20.2 I do not see a way to limit
the system from running mappers and reducers together with the total
exceeding individual limits.
I find that when my mappers are about 50% done the system kicks off
reducers. I have raised the maxmemory in
chilld.java,vm.opts because I have been hitting GC limits and the values
work well when I am running 6 mappers OR
6 reducers but when my mappers are half way done I see 6 mappers AND 6
reducers running and this challenges the
total memory on a node.
How can I keep the total tasks on a node under control without limiting the
maximum mappers and reducers to half the total I want??
--
Steven M. Lewis PhD
4221 105th Ave NE
Kirkland, WA 98033
206-384-1340 (cell)
Skype lordjoe_com
Re: Any way to limit the total tasks running on a node in 0.202
Posted by Harsh J <ha...@cloudera.com>.
Steve,
Perhaps raise your reducer slowstart config to wait till about 90% of
a Job's mappers complete, such that they may start later (almost when
all map cycles are done?)
The prop name is mapred.reduce.slowstart.completed.maps, and the default is 10%.
On Thu, Nov 10, 2011 at 8:36 AM, Steve Lewis <lo...@gmail.com> wrote:
> Hadoop can set the maximum mappers and reducers running on a node but under
> 0.20.2 I do not see a way to limit
> the system from running mappers and reducers together with the total
> exceeding individual limits.
> I find that when my mappers are about 50% done the system kicks off
> reducers. I have raised the maxmemory in
> chilld.java,vm.opts because I have been hitting GC limits and the values
> work well when I am running 6 mappers OR
> 6 reducers but when my mappers are half way done I see 6 mappers AND 6
> reducers running and this challenges the
> total memory on a node.
> How can I keep the total tasks on a node under control without limiting the
> maximum mappers and reducers to half the total I want??
> --
> Steven M. Lewis PhD
> 4221 105th Ave NE
> Kirkland, WA 98033
> 206-384-1340 (cell)
> Skype lordjoe_com
>
>
>
--
Harsh J