Profiling Map/Reduce Tasks

[Richa Khandelwal <ri...@gmail.com>]

Hi,
Does Map/Reduce profiles jobs down to milliseconds. From what I can see in
the logs, there is no time specified for the job. Although CPU TIME is an
information that should be present in the logs, it was not profiled and the
response time can only be noted in down to seconds from the runtime progress
of the jobs.

Does someone know how to efficiently profile map reduce jobs?

Thanks,
Richa Khandelwal


University Of California,
Santa Cruz.
Ph:425-241-7763

Re: Profiling Map/Reduce Tasks

[Chris Douglas <ch...@yahoo-inc.com>]

I use YourKit (http://yourkit.com/).

You'll also want to look at the following parameters:

   mapred.task.profile.params (e.g. - 
agentlib:yjpagent=sampling=onexit=snapshot,dir=%s)
   mapred.task.profile  (true)
   mapred.task.profile.reduces (0-2 (or whatever))
   mapred.task.profile.maps (0-2)

By default, YK filters org.apache.*, so be sure to disable that. It's  
simplest to profile individual tasks in psudo-distributed mode, though  
clearly that has its drawbacks. -C

On Mar 9, 2009, at 12:05 AM, Rasit OZDAS wrote:

> I note System.currentTimeMillis() at the beginning of main function,
> then at the end I use a while loop to wait for the job,
>
> while (!runningJob.isComplete())
>      Thread.sleep(1000);
>
> Then again I note the system time. But this only gives the total  
> amount of
> time passed.
>
> Rasit
>
> 2009/3/8 Richa Khandelwal <ri...@gmail.com>
>
>> Hi,
>> Does Map/Reduce profiles jobs down to milliseconds. From what I can  
>> see in
>> the logs, there is no time specified for the job. Although CPU TIME  
>> is an
>> information that should be present in the logs, it was not profiled  
>> and the
>> response time can only be noted in down to seconds from the runtime
>> progress
>> of the jobs.
>>
>> Does someone know how to efficiently profile map reduce jobs?
>>
>> Thanks,
>> Richa Khandelwal
>>
>>
>> University Of California,
>> Santa Cruz.
>> Ph:425-241-7763
>>
>
>
>
> -- 
> M. Raşit ÖZDAŞ

Re: Profiling Map/Reduce Tasks

[Rasit OZDAS <ra...@gmail.com>]

I note System.currentTimeMillis() at the beginning of main function,
then at the end I use a while loop to wait for the job,

while (!runningJob.isComplete())
      Thread.sleep(1000);

Then again I note the system time. But this only gives the total amount of
time passed.

Rasit

2009/3/8 Richa Khandelwal <ri...@gmail.com>

> Hi,
> Does Map/Reduce profiles jobs down to milliseconds. From what I can see in
> the logs, there is no time specified for the job. Although CPU TIME is an
> information that should be present in the logs, it was not profiled and the
> response time can only be noted in down to seconds from the runtime
> progress
> of the jobs.
>
> Does someone know how to efficiently profile map reduce jobs?
>
> Thanks,
> Richa Khandelwal
>
>
> University Of California,
> Santa Cruz.
> Ph:425-241-7763
>



-- 
M. Raşit ÖZDAŞ