Custom Hadoop metrics in Map Reduce jobs

[Dino Kečo <di...@gmail.com>]

Hi Hadoop users,

I have been reading about Hadoop metrics framework, and I was wondering is
it possible to create custom metrics for specific job.

In my use case I want to capture some specific information per record which
is processed. So I was thinking to create custom code in my Hadoop job to
send these information using Hadoop metrics. Is this possible to implement
? If so can anyone share any example how things should be done?

Thanks,
dinok

Re: Custom Hadoop metrics in Map Reduce jobs

[Harsh J <ha...@cloudera.com>]

Jobs provide counters as a way to store some information. Are you
looking for something beyond plain counters?

On Wed, Nov 2, 2011 at 1:14 PM, Dino Kečo <di...@gmail.com> wrote:
> Hi Hadoop users,
> I have been reading about Hadoop metrics framework, and I was wondering is
> it possible to create custom metrics for specific job.
> In my use case I want to capture some specific information per record which
> is processed. So I was thinking to create custom code in my Hadoop job to
> send these information using Hadoop metrics. Is this possible to implement ?
> If so can anyone share any example how things should be done?
> Thanks,
> dinok



-- 
Harsh J

Re: Custom Hadoop metrics in Map Reduce jobs

[Dino Kečo <di...@gmail.com>]

Hi Harsh,

Yes I am looking for much detailed information beyond counters. Counters
are pretty cool but I would like to gather more information related to
record (like processing time in some part of code, some business logic
attributes and etc.) So I was hoping to use hadoop metrics to save that
into file or table and use it for reporting.

Daniel,

Thanks for info, please share your blog links so we can see something new :)

Regards,
dinok


On Wed, Nov 2, 2011 at 10:59 AM, Daniel Iancu <da...@1and1.ro> wrote:

> **
> Hi Dino
> It can't be used effectively to monitor MR mappers since tasks on a node
> run as separate JVMs and each task will overwrite the same metric published
> by other tasks. You cannot aggregate values from running tasks before
> publish them because tasks don't communicate with each other.
> If you publish a metric per task, this will give you tones of graphs in
> Ganglia.
>
> On other hand I extended it the API and use it to monitor our client
> (webapp) and it works fine and provide valuable information. I have started
> a blog post to document this, will be ready soon, I'll add the link here if
> you need some examples. I wonder why this API is not a separate project, it
> can be reused in apps that don't need Hadoop. I could not find any java
> client API to publish metrics in Ganglia.
>
> Regards
> Daniel
>
>
>
>
> On 11/02/2011 09:44 AM, Dino Kečo wrote:
>
> Hi Hadoop users,
>
>  I have been reading about Hadoop metrics framework, and I was wondering
> is it possible to create custom metrics for specific job.
>
>  In my use case I want to capture some specific information per record
> which is processed. So I was thinking to create custom code in my Hadoop
> job to send these information using Hadoop metrics. Is this possible to
> implement ? If so can anyone share any example how things should be done?
>
>  Thanks,
> dinok
>
>
>

Re: Custom Hadoop metrics in Map Reduce jobs

[Daniel Iancu <da...@1and1.ro>]

Hi Dino
It can't be used effectively to monitor MR mappers since tasks on a node 
run as separate JVMs and each task will overwrite the same metric 
published by other tasks. You cannot aggregate values from running tasks 
before publish them because tasks don't communicate with each other.
If you publish a metric per task, this will give you tones of graphs in 
Ganglia.

On other hand I extended it the API and use it to monitor our client 
(webapp) and it works fine and provide valuable information. I have 
started a blog post to document this, will be ready soon, I'll add the 
link here if you need some examples. I wonder why this API is not a 
separate project, it can be reused in apps that don't need Hadoop. I 
could not find any java client API to publish metrics in Ganglia.

Regards
Daniel



On 11/02/2011 09:44 AM, Dino Kečo wrote:
> Hi Hadoop users,
>
> I have been reading about Hadoop metrics framework, and I was 
> wondering is it possible to create custom metrics for specific job.
>
> In my use case I want to capture some specific information per record 
> which is processed. So I was thinking to create custom code in my 
> Hadoop job to send these information using Hadoop metrics. Is this 
> possible to implement ? If so can anyone share any example how things 
> should be done?
>
> Thanks,
> dinok

Re: Custom Hadoop metrics in Map Reduce jobs

[David Rosenstrauch <da...@darose.net>]

On 11/02/2011 03:44 AM, Dino Kečo wrote:
> I have been reading about Hadoop metrics framework, and I was wondering is
> it possible to create custom metrics for specific job.

You can create your own custom counters, to count particular statistics 
you're interested in.  That can take you a long way in this regard.

HTH,

DR