Custom reduce progress reporting

[Ed Mazur <ma...@cs.umass.edu>]

Is there a configuration property or some other mechanism for changing
how reduce tasks report progress? E.g., instead of the shuffle, merge,
and reduce each counting for 33%, can this be changed?

The Hadoop Online Prototype did this, but I can't figure out how.

Ed

Re: Custom reduce progress reporting

[Ravi Gummadi <gr...@yahoo-inc.com>]

Currently, they are hardcoded in sourcecode. Please see 
Progress.addPhase() and Progress.addPhase(String status, float 
weightage) and their calls in MapTask.run() and ReduceTask.run() for 
changing the weightages of phases. In ReduceTask.run(), change the calls 
from addPhase() to addPhase(status, weightage) for all 3 phases with 
different weightages(like 0.4, 0.25, 0.35 for the 3 phases).

-Ravi

Ed Mazur wrote:
> Is there a configuration property or some other mechanism for changing
> how reduce tasks report progress? E.g., instead of the shuffle, merge,
> and reduce each counting for 33%, can this be changed?
>
> The Hadoop Online Prototype did this, but I can't figure out how.
>
> Ed
>