You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by Taeho Kang <tk...@gmail.com> on 2009/02/09 08:26:15 UTC
Reduce won't start until Map stage reaches 100%?
Dear All,
With Hadoop 0.19.0, Reduce stage does not start until Map stage gets to the
100% completion.
Has anyone faced the similar situation?
... ...
- map 90% reduce 0%
- map 91% reduce 0%
- map 92% reduce 0%
- map 93% reduce 0%
- map 94% reduce 0%
- map 95% reduce 0%
- map 96% reduce 0%
- map 97% reduce 0%
- map 98% reduce 0%
- map 99% reduce 0%
- map 100% reduce 0%
- map 100% reduce 1%
- map 100% reduce 2%
- map 100% reduce 3%
- map 100% reduce 4%
- map 100% reduce 5%
- map 100% reduce 6%
- map 100% reduce 7%
- map 100% reduce 8%
- map 100% reduce 9%
Thank you all in advance,
/Taeho
Re: Reduce won't start until Map stage reaches 100%?
Posted by Arun C Murthy <ac...@yahoo-inc.com>.
On Feb 8, 2009, at 11:26 PM, Taeho Kang wrote:
> Dear All,
>
> With Hadoop 0.19.0, Reduce stage does not start until Map stage gets
> to the
> 100% completion.
> Has anyone faced the similar situation?
>
How many maps and reduces does your job have?
Arun
Re: Reduce won't start until Map stage reaches 100%?
Posted by Matei Zaharia <ma...@cloudera.com>.
I believe that in Hadoop 0.19, scheduling was changed so that reduces don't
start until 5% of maps have completed. The reasoning for this is that
reduces can't do anything until there is some map output to copy over the
network. So, if your job has very few map tasks, you won't see reduces start
until the end.
On Mon, Feb 9, 2009 at 12:51 AM, zhuweimin <xi...@tsm.kddilabs.jp> wrote:
> Hi
>
> I think the number of your job's reduce task is 1
> because if the number of reduce task is 1 then reduce stage does not start
> until Map stage 100% completion.
>
> zhuweimin
>
> -----Original Message-----
> From: Taeho Kang [mailto:tkang1@gmail.com]
> Sent: Monday, February 09, 2009 4:26 PM
> To: hadoop-user@lucene.apache.org
> Subject: Reduce won't start until Map stage reaches 100%?
>
> Dear All,
>
> With Hadoop 0.19.0, Reduce stage does not start until Map stage gets to the
> 100% completion.
> Has anyone faced the similar situation?
>
> ... ...
> - map 90% reduce 0%
> - map 91% reduce 0%
> - map 92% reduce 0%
> - map 93% reduce 0%
> - map 94% reduce 0%
> - map 95% reduce 0%
> - map 96% reduce 0%
> - map 97% reduce 0%
> - map 98% reduce 0%
> - map 99% reduce 0%
> - map 100% reduce 0%
> - map 100% reduce 1%
> - map 100% reduce 2%
> - map 100% reduce 3%
> - map 100% reduce 4%
> - map 100% reduce 5%
> - map 100% reduce 6%
> - map 100% reduce 7%
> - map 100% reduce 8%
> - map 100% reduce 9%
>
> Thank you all in advance,
>
> /Taeho
>
>
>
RE: Reduce won't start until Map stage reaches 100%?
Posted by zhuweimin <xi...@tsm.kddilabs.jp>.
Hi
I think the number of your job's reduce task is 1
because if the number of reduce task is 1 then reduce stage does not start
until Map stage 100% completion.
zhuweimin
-----Original Message-----
From: Taeho Kang [mailto:tkang1@gmail.com]
Sent: Monday, February 09, 2009 4:26 PM
To: hadoop-user@lucene.apache.org
Subject: Reduce won't start until Map stage reaches 100%?
Dear All,
With Hadoop 0.19.0, Reduce stage does not start until Map stage gets to the
100% completion.
Has anyone faced the similar situation?
... ...
- map 90% reduce 0%
- map 91% reduce 0%
- map 92% reduce 0%
- map 93% reduce 0%
- map 94% reduce 0%
- map 95% reduce 0%
- map 96% reduce 0%
- map 97% reduce 0%
- map 98% reduce 0%
- map 99% reduce 0%
- map 100% reduce 0%
- map 100% reduce 1%
- map 100% reduce 2%
- map 100% reduce 3%
- map 100% reduce 4%
- map 100% reduce 5%
- map 100% reduce 6%
- map 100% reduce 7%
- map 100% reduce 8%
- map 100% reduce 9%
Thank you all in advance,
/Taeho
Re: Reduce won't start until Map stage reaches 100%?
Posted by james warren <ja...@rockyou.com>.
Hi Taeho -
That definitely isn't the typical behavior. Could you provide more details
about your application?
On Sun, Feb 8, 2009 at 11:26 PM, Taeho Kang <tk...@gmail.com> wrote:
> Dear All,
>
> With Hadoop 0.19.0, Reduce stage does not start until Map stage gets to the
> 100% completion.
> Has anyone faced the similar situation?
>
> ... ...
> - map 90% reduce 0%
> - map 91% reduce 0%
> - map 92% reduce 0%
> - map 93% reduce 0%
> - map 94% reduce 0%
> - map 95% reduce 0%
> - map 96% reduce 0%
> - map 97% reduce 0%
> - map 98% reduce 0%
> - map 99% reduce 0%
> - map 100% reduce 0%
> - map 100% reduce 1%
> - map 100% reduce 2%
> - map 100% reduce 3%
> - map 100% reduce 4%
> - map 100% reduce 5%
> - map 100% reduce 6%
> - map 100% reduce 7%
> - map 100% reduce 8%
> - map 100% reduce 9%
>
> Thank you all in advance,
>
> /Taeho
>