Re: When a Reduce Task starts?

[Arun C Murthy <ac...@yahoo-inc.com>]

On Dec 23, 2010, at 9:20 PM, pig wrote:
> For some special reduce jobs that do not rely on the order of  
> (key,value) pairs,  the sort phase is of no use.
> In this situation, theoretically speaking, reduce can be started  
> before all of the map task finished.
> But why hadoop doesn't support this feature? For example, it may be  
> specified as an argument when committing a job.
>

Several reasons...

A major problem is errors - a map may fail after it's output has been  
'shuffled' by some reduces, not all (i.e. copied by some reduces). In  
this case, it's really hard to track and discard duplicate key/value  
pairs.

The behaviour you seek is quite easy to model by running map-only  
jobs, saving their output to HDFS and processing in the next job -  
albeit with some performance penalties. But, this keeps the MR  
framework very simple and stable.

Arun