You are viewing a plain text version of this content. The canonical link for it is here.
Posted to common-user@hadoop.apache.org by biblio84 <fa...@gmail.com> on 2010/06/20 11:19:45 UTC
Re: Reduce starts as Map finishes
Same is the problem with me. Why does reducer start working before mappers
are done? Have u found the answer?
--
View this message in context: http://old.nabble.com/Reduce-starts-as-Map-finishes-tp27662411p28939648.html
Sent from the Hadoop core-user mailing list archive at Nabble.com.
Re: Reduce starts as Map finishes
Posted by 李钰 <ca...@gmail.com>.
Hi Eric,
Here I've got a question: why the
mapred.reduce.slowstart.completed.maps parameter is added in 0.20.x release?
What's the designing thought? I found sometimes setting this parameter do
improve performance, but not quite clear about why. Do you know under what
conditions should this parameter be used and play a role? Thanks and looking
forward to your reply.
Best Regards,
Carp
2010/6/21 Eric Sammer <es...@cloudera.com>
> The reducers will start the copy phase while mappers are still running
> but they will not call the user reduce method until all mappers are
> complete. This is an optimization that shouldn't matter to the user.
>
> Eric
>
> On Jun 20, 2010, at 8:11 AM, biblio84 <fa...@gmail.com> wrote:
>
> >
> > Same is the problem with me. Why does reducer start working before
> > mappers
> > are done? Have u found the answer?
> > --
> > View this message in context:
> http://old.nabble.com/Reduce-starts-as-Map-finishes-tp27662411p28939648.html
> > Sent from the Hadoop core-user mailing list archive at Nabble.com.
> >
>
Re: Reduce starts as Map finishes
Posted by Eric Sammer <es...@cloudera.com>.
The reducers will start the copy phase while mappers are still running
but they will not call the user reduce method until all mappers are
complete. This is an optimization that shouldn't matter to the user.
Eric
On Jun 20, 2010, at 8:11 AM, biblio84 <fa...@gmail.com> wrote:
>
> Same is the problem with me. Why does reducer start working before
> mappers
> are done? Have u found the answer?
> --
> View this message in context: http://old.nabble.com/Reduce-starts-as-Map-finishes-tp27662411p28939648.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
Re: Reduce starts as Map finishes
Posted by Ted Yu <yu...@gmail.com>.
>From Allen's earlier email:
mapred.reduce.slowstart.
completed.maps is the setting you are looking for.
See also https://issues.apache.org/jira/browse/MAPREDUCE-1184
On Sun, Jun 20, 2010 at 2:19 AM, biblio84 <fa...@gmail.com> wrote:
>
> Same is the problem with me. Why does reducer start working before mappers
> are done? Have u found the answer?
> --
> View this message in context:
> http://old.nabble.com/Reduce-starts-as-Map-finishes-tp27662411p28939648.html
> Sent from the Hadoop core-user mailing list archive at Nabble.com.
>
>