Re: Measuring Shuffle time for MR job

[Thiago Vieira <tp...@gmail.com>]

Sorry about this late response, but Apache Rumen(
http://hadoop.apache.org/docs/mapreduce/r0.22.0/rumen.html) is a good
solution to extract several informations from job log.

--
Thiago Vieira


On Mon, Aug 27, 2012 at 10:54 AM, Raj Vishwanathan <ra...@yahoo.com>wrote:

> You can extract the shuffle time from the job log.
>
> Take a look at
>
> https://github.com/rajvish/hadoop-summary
>
>
> Raj
>
>
>
> >________________________________
> > From: Bertrand Dechoux <de...@gmail.com>
> >To: common-user@hadoop.apache.org
> >Sent: Monday, August 27, 2012 12:57 AM
> >Subject: Re: Measuring Shuffle time for MR job
> >
> >Shuffle time is considered as part of the reduce step. Without reduce,
> >there is no need for shuffling.
> >One way to measure it would be using the full reduce time with a
> >'/dev/null' reducer.
> >
> >I am not aware of any way to measure it.
> >
> >Regards
> >
> >Bertrand
> >
> >On Mon, Aug 27, 2012 at 8:18 AM, praveenesh kumar <praveenesh@gmail.com
> >wrote:
> >
> >> Is there a way to know the total shuffle time of a map-reduce job - I
> mean
> >> some command or output  that can tell that ?
> >>
> >> I want to measure total map, total shuffle and total reduce time for my
> MR
> >> job -- how can I achieve it ? I am using hadoop 0.20.205
> >>
> >>
> >> Regards,
> >> Praveenesh
> >>
> >
> >
> >
> >--
> >Bertrand Dechoux
> >
> >
> >
>