Measuring time in a Hadoop MapReduce job.

[Amit Sangroya <sa...@gmail.com>]

Hi,

I am trying to monitor the time to complete a map phase and reduce
phase in hadoop. Is there any way to measure the time taken to
complete map and reduce phase in a cluster.

Thanks,
Amit

Re: Measuring time in a Hadoop MapReduce job.

[Amit Sangroya <sa...@gmail.com>]

Dear Keren,

Thanks for your response!

I understood from your response that I can parse the job history file
and extract the information about START_TIME and FINISH_TIME.

Would it be possible to know about a simpler way where I know a
MAP_PHASE_START_TIME and MAP_PHASE_FINISH_TIME and similarly
REDUCE_PHASE_START_TIME and REDUCE_PHASE_FINISH_TIME.


Thanks,
Amit



On Mon, Jul 4, 2011 at 11:12 AM, Keren Ouaknine <ke...@gmail.com> wrote:
> Hello Amit,
>
> There are several ways, the one I am currently using for measurements is
> based on the job history file.
> This file has a line for each event, thus the last map event (indicating on
> the completion of the map task) would be the last record in time with both
> string "MAP" and "TASK_FINISHED". Similarly for the last reduce event: look
> for the last record in time with both string "REDUCE" and "TASK_FINISHED".
>
> If you find a better way, please update me!
>
> Thanks,
> Keren
>
>
> On Mon, Jul 4, 2011 at 12:04 PM, Amit Sangroya <sa...@gmail.com>wrote:
>
>> Hi,
>>
>> I am trying to monitor the time to complete a map phase and reduce
>> phase in hadoop. Is there any way to measure the time taken to
>> complete map and reduce phase in a cluster.
>>
>> Thanks,
>> Amit
>>
>
>
>
> --
> Keren Ouaknine
> Cell: +972 54 2565404
> Web: www.kereno.com
>

Re: Measuring time in a Hadoop MapReduce job.

[Keren Ouaknine <ke...@gmail.com>]

Hello Amit,

There are several ways, the one I am currently using for measurements is
based on the job history file.
This file has a line for each event, thus the last map event (indicating on
the completion of the map task) would be the last record in time with both
string "MAP" and "TASK_FINISHED". Similarly for the last reduce event: look
for the last record in time with both string "REDUCE" and "TASK_FINISHED".

If you find a better way, please update me!

Thanks,
Keren


On Mon, Jul 4, 2011 at 12:04 PM, Amit Sangroya <sa...@gmail.com>wrote:

> Hi,
>
> I am trying to monitor the time to complete a map phase and reduce
> phase in hadoop. Is there any way to measure the time taken to
> complete map and reduce phase in a cluster.
>
> Thanks,
> Amit
>



-- 
Keren Ouaknine
Cell: +972 54 2565404
Web: www.kereno.com