You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@flume.apache.org by Ashish <pa...@gmail.com> on 2014/11/09 05:21:06 UTC
Using CodaHale metrics for monitoring
Hi,
Have hacked a bit into our existing instrumentation package and piggy
backed cohahale metrics package. Here is one sample for Spooled
Directory source (with instrumentation only for Source and Channel ),
using console reporter
-- Gauges ----------------------------------------------------------------------
org.apache.flume.instrumentation.ChannelCounter.channel.current.size
value = 200
org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
value = 2.0
org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
value = 0
-- Counters --------------------------------------------------------------------
org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
count = 1138800
org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
count = 1138800
org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
count = 1138601
org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
count = 1138600
org.apache.flume.instrumentation.SourceCounter.src.events.accepted
count = 1138800
org.apache.flume.instrumentation.SourceCounter.src.events.received
count = 1138800
src.append-batch.accepted
count = 11388
src.append-batch.received
count = 11388
src.append.accepted
count = 0
src.append.received
count = 0
-- Meters ----------------------------------------------------------------------
eventAcceptedMeter
count = 1138800
mean rate = 106478.56 events/second
1-minute rate = 93420.18 events/second
5-minute rate = 91389.24 events/second
15-minute rate = 91037.40 events/second
eventReceivedMeter
count = 1138800
mean rate = 106462.14 events/second
1-minute rate = 93420.18 events/second
5-minute rate = 91389.24 events/second
15-minute rate = 91037.40 events/second
If there is interest in the community, can raise a jira and continue
to work on it.
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Hari Shreedharan <hs...@cloudera.com>.
Makes sense.
Thanks,
Hari
On Tue, Nov 11, 2014 at 9:51 PM, Ashish <pa...@gmail.com> wrote:
> That's what I had in mind, just need to figure out how to do it with
> CodaHale. Want to avoid CodaHale MetricRegistry class floating around
> in code. 1st step would be to migrate to CodaHale, I am yet to figure
> out how managing the Reporter part. Let me try to put a patch
> together.
> On Wed, Nov 12, 2014 at 9:29 AM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>> Currently the metrics exposed via Sources/sinks/channels are hardcoded. We could add a method to add custom metrics (I think it is already there in MonitoredCounterGroup, just not in the *Counter classes).
>>
>>
>> Thanks,
>> Hari
>>
>> On Tue, Nov 11, 2014 at 6:34 PM, Ashish <pa...@gmail.com> wrote:
>>
>>> I was just playing around so piggy-backed it. The changes have to be
>>> within MonitoredCounterGroup and its sub-classes, as its used across
>>> the implementations. Let me raise a JIRA for this.
>>> One thing that I would like to have in future is to allow
>>> Sinks/Channels/Sources to be able to have custom counters/Meters and
>>> be available to have those Metrics available via the framework. I
>>> think CounterGroup was created for the same purpose.
>>> Let me create a JIRA for this and work on it.
>>> thanks
>>> ashish
>>> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
>>> <hs...@cloudera.com> wrote:
>>>> I think it is best to replace the current system rather than having multiple metrics systems. We should implement it such that it is transparent to user code, and the changes only go into MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Mon, Nov 10, 2014 at 5:36 PM, Ashish <pa...@gmail.com> wrote:
>>>>
>>>>> Codahale does have lot more features like the rate/sec one, which is
>>>>> most needed metrics. It provides a lot of reporters out of box
>>>>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>>>>> enable it, rather than writing custom components.
>>>>> As of now we have to hide it within MonitoredCounterGroup and its
>>>>> sub-classes, that's what I did for running it. It was running parallel
>>>>> to existing system. Let me work more on it and see if I can simplify
>>>>> the same.
>>>>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>>>>> <hs...@cloudera.com> wrote:
>>>>>> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>>>>>
>>>>>>
>>>>>> Thanks,
>>>>>> Hari
>>>>>>
>>>>>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>> Have hacked a bit into our existing instrumentation package and piggy
>>>>>>> backed cohahale metrics package. Here is one sample for Spooled
>>>>>>> Directory source (with instrumentation only for Source and Channel ),
>>>>>>> using console reporter
>>>>>>> -- Gauges ----------------------------------------------------------------------
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>>>>> value = 200
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>>>>> value = 2.0
>>>>>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>>>>> value = 0
>>>>>>> -- Counters --------------------------------------------------------------------
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>>>>> count = 1138800
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>>>>> count = 1138800
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>>>>> count = 1138601
>>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>>>>> count = 1138600
>>>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>>>>> count = 1138800
>>>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>>>>> count = 1138800
>>>>>>> src.append-batch.accepted
>>>>>>> count = 11388
>>>>>>> src.append-batch.received
>>>>>>> count = 11388
>>>>>>> src.append.accepted
>>>>>>> count = 0
>>>>>>> src.append.received
>>>>>>> count = 0
>>>>>>> -- Meters ----------------------------------------------------------------------
>>>>>>> eventAcceptedMeter
>>>>>>> count = 1138800
>>>>>>> mean rate = 106478.56 events/second
>>>>>>> 1-minute rate = 93420.18 events/second
>>>>>>> 5-minute rate = 91389.24 events/second
>>>>>>> 15-minute rate = 91037.40 events/second
>>>>>>> eventReceivedMeter
>>>>>>> count = 1138800
>>>>>>> mean rate = 106462.14 events/second
>>>>>>> 1-minute rate = 93420.18 events/second
>>>>>>> 5-minute rate = 91389.24 events/second
>>>>>>> 15-minute rate = 91037.40 events/second
>>>>>>> If there is interest in the community, can raise a jira and continue
>>>>>>> to work on it.
>>>>>>> --
>>>>>>> thanks
>>>>>>> ashish
>>>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>>> --
>>>>> thanks
>>>>> ashish
>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>> --
>>> thanks
>>> ashish
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Ashish <pa...@gmail.com>.
That's what I had in mind, just need to figure out how to do it with
CodaHale. Want to avoid CodaHale MetricRegistry class floating around
in code. 1st step would be to migrate to CodaHale, I am yet to figure
out how managing the Reporter part. Let me try to put a patch
together.
On Wed, Nov 12, 2014 at 9:29 AM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> Currently the metrics exposed via Sources/sinks/channels are hardcoded. We could add a method to add custom metrics (I think it is already there in MonitoredCounterGroup, just not in the *Counter classes).
>
>
> Thanks,
> Hari
>
> On Tue, Nov 11, 2014 at 6:34 PM, Ashish <pa...@gmail.com> wrote:
>
>> I was just playing around so piggy-backed it. The changes have to be
>> within MonitoredCounterGroup and its sub-classes, as its used across
>> the implementations. Let me raise a JIRA for this.
>> One thing that I would like to have in future is to allow
>> Sinks/Channels/Sources to be able to have custom counters/Meters and
>> be available to have those Metrics available via the framework. I
>> think CounterGroup was created for the same purpose.
>> Let me create a JIRA for this and work on it.
>> thanks
>> ashish
>> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>>> I think it is best to replace the current system rather than having multiple metrics systems. We should implement it such that it is transparent to user code, and the changes only go into MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Mon, Nov 10, 2014 at 5:36 PM, Ashish <pa...@gmail.com> wrote:
>>>
>>>> Codahale does have lot more features like the rate/sec one, which is
>>>> most needed metrics. It provides a lot of reporters out of box
>>>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>>>> enable it, rather than writing custom components.
>>>> As of now we have to hide it within MonitoredCounterGroup and its
>>>> sub-classes, that's what I did for running it. It was running parallel
>>>> to existing system. Let me work more on it and see if I can simplify
>>>> the same.
>>>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>>>> <hs...@cloudera.com> wrote:
>>>>> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>>>>
>>>>>
>>>>> Thanks,
>>>>> Hari
>>>>>
>>>>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>> Have hacked a bit into our existing instrumentation package and piggy
>>>>>> backed cohahale metrics package. Here is one sample for Spooled
>>>>>> Directory source (with instrumentation only for Source and Channel ),
>>>>>> using console reporter
>>>>>> -- Gauges ----------------------------------------------------------------------
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>>>> value = 200
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>>>> value = 2.0
>>>>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>>>> value = 0
>>>>>> -- Counters --------------------------------------------------------------------
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>>>> count = 1138800
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>>>> count = 1138800
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>>>> count = 1138601
>>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>>>> count = 1138600
>>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>>>> count = 1138800
>>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>>>> count = 1138800
>>>>>> src.append-batch.accepted
>>>>>> count = 11388
>>>>>> src.append-batch.received
>>>>>> count = 11388
>>>>>> src.append.accepted
>>>>>> count = 0
>>>>>> src.append.received
>>>>>> count = 0
>>>>>> -- Meters ----------------------------------------------------------------------
>>>>>> eventAcceptedMeter
>>>>>> count = 1138800
>>>>>> mean rate = 106478.56 events/second
>>>>>> 1-minute rate = 93420.18 events/second
>>>>>> 5-minute rate = 91389.24 events/second
>>>>>> 15-minute rate = 91037.40 events/second
>>>>>> eventReceivedMeter
>>>>>> count = 1138800
>>>>>> mean rate = 106462.14 events/second
>>>>>> 1-minute rate = 93420.18 events/second
>>>>>> 5-minute rate = 91389.24 events/second
>>>>>> 15-minute rate = 91037.40 events/second
>>>>>> If there is interest in the community, can raise a jira and continue
>>>>>> to work on it.
>>>>>> --
>>>>>> thanks
>>>>>> ashish
>>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>>> --
>>>> thanks
>>>> ashish
>>>> Blog: http://www.ashishpaliwal.com/blog
>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> --
>> thanks
>> ashish
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Hari Shreedharan <hs...@cloudera.com>.
Currently the metrics exposed via Sources/sinks/channels are hardcoded. We could add a method to add custom metrics (I think it is already there in MonitoredCounterGroup, just not in the *Counter classes).
Thanks,
Hari
On Tue, Nov 11, 2014 at 6:34 PM, Ashish <pa...@gmail.com> wrote:
> I was just playing around so piggy-backed it. The changes have to be
> within MonitoredCounterGroup and its sub-classes, as its used across
> the implementations. Let me raise a JIRA for this.
> One thing that I would like to have in future is to allow
> Sinks/Channels/Sources to be able to have custom counters/Meters and
> be available to have those Metrics available via the framework. I
> think CounterGroup was created for the same purpose.
> Let me create a JIRA for this and work on it.
> thanks
> ashish
> On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>> I think it is best to replace the current system rather than having multiple metrics systems. We should implement it such that it is transparent to user code, and the changes only go into MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
>>
>>
>> Thanks,
>> Hari
>>
>> On Mon, Nov 10, 2014 at 5:36 PM, Ashish <pa...@gmail.com> wrote:
>>
>>> Codahale does have lot more features like the rate/sec one, which is
>>> most needed metrics. It provides a lot of reporters out of box
>>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>>> enable it, rather than writing custom components.
>>> As of now we have to hide it within MonitoredCounterGroup and its
>>> sub-classes, that's what I did for running it. It was running parallel
>>> to existing system. Let me work more on it and see if I can simplify
>>> the same.
>>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>>> <hs...@cloudera.com> wrote:
>>>> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>>>
>>>>
>>>> Thanks,
>>>> Hari
>>>>
>>>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>> Have hacked a bit into our existing instrumentation package and piggy
>>>>> backed cohahale metrics package. Here is one sample for Spooled
>>>>> Directory source (with instrumentation only for Source and Channel ),
>>>>> using console reporter
>>>>> -- Gauges ----------------------------------------------------------------------
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>>> value = 200
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>>> value = 2.0
>>>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>>> value = 0
>>>>> -- Counters --------------------------------------------------------------------
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>>> count = 1138800
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>>> count = 1138800
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>>> count = 1138601
>>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>>> count = 1138600
>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>>> count = 1138800
>>>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>>> count = 1138800
>>>>> src.append-batch.accepted
>>>>> count = 11388
>>>>> src.append-batch.received
>>>>> count = 11388
>>>>> src.append.accepted
>>>>> count = 0
>>>>> src.append.received
>>>>> count = 0
>>>>> -- Meters ----------------------------------------------------------------------
>>>>> eventAcceptedMeter
>>>>> count = 1138800
>>>>> mean rate = 106478.56 events/second
>>>>> 1-minute rate = 93420.18 events/second
>>>>> 5-minute rate = 91389.24 events/second
>>>>> 15-minute rate = 91037.40 events/second
>>>>> eventReceivedMeter
>>>>> count = 1138800
>>>>> mean rate = 106462.14 events/second
>>>>> 1-minute rate = 93420.18 events/second
>>>>> 5-minute rate = 91389.24 events/second
>>>>> 15-minute rate = 91037.40 events/second
>>>>> If there is interest in the community, can raise a jira and continue
>>>>> to work on it.
>>>>> --
>>>>> thanks
>>>>> ashish
>>>>> Blog: http://www.ashishpaliwal.com/blog
>>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>>> --
>>> thanks
>>> ashish
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Ashish <pa...@gmail.com>.
I was just playing around so piggy-backed it. The changes have to be
within MonitoredCounterGroup and its sub-classes, as its used across
the implementations. Let me raise a JIRA for this.
One thing that I would like to have in future is to allow
Sinks/Channels/Sources to be able to have custom counters/Meters and
be available to have those Metrics available via the framework. I
think CounterGroup was created for the same purpose.
Let me create a JIRA for this and work on it.
thanks
ashish
On Wed, Nov 12, 2014 at 1:11 AM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> I think it is best to replace the current system rather than having multiple metrics systems. We should implement it such that it is transparent to user code, and the changes only go into MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
>
>
> Thanks,
> Hari
>
> On Mon, Nov 10, 2014 at 5:36 PM, Ashish <pa...@gmail.com> wrote:
>
>> Codahale does have lot more features like the rate/sec one, which is
>> most needed metrics. It provides a lot of reporters out of box
>> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
>> enable it, rather than writing custom components.
>> As of now we have to hide it within MonitoredCounterGroup and its
>> sub-classes, that's what I did for running it. It was running parallel
>> to existing system. Let me work more on it and see if I can simplify
>> the same.
>> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
>> <hs...@cloudera.com> wrote:
>>> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>>
>>>
>>> Thanks,
>>> Hari
>>>
>>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>>>
>>>> Hi,
>>>> Have hacked a bit into our existing instrumentation package and piggy
>>>> backed cohahale metrics package. Here is one sample for Spooled
>>>> Directory source (with instrumentation only for Source and Channel ),
>>>> using console reporter
>>>> -- Gauges ----------------------------------------------------------------------
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>>> value = 200
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>>> value = 2.0
>>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>>> value = 0
>>>> -- Counters --------------------------------------------------------------------
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>>> count = 1138800
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>>> count = 1138800
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>>> count = 1138601
>>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>>> count = 1138600
>>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>>> count = 1138800
>>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>>> count = 1138800
>>>> src.append-batch.accepted
>>>> count = 11388
>>>> src.append-batch.received
>>>> count = 11388
>>>> src.append.accepted
>>>> count = 0
>>>> src.append.received
>>>> count = 0
>>>> -- Meters ----------------------------------------------------------------------
>>>> eventAcceptedMeter
>>>> count = 1138800
>>>> mean rate = 106478.56 events/second
>>>> 1-minute rate = 93420.18 events/second
>>>> 5-minute rate = 91389.24 events/second
>>>> 15-minute rate = 91037.40 events/second
>>>> eventReceivedMeter
>>>> count = 1138800
>>>> mean rate = 106462.14 events/second
>>>> 1-minute rate = 93420.18 events/second
>>>> 5-minute rate = 91389.24 events/second
>>>> 15-minute rate = 91037.40 events/second
>>>> If there is interest in the community, can raise a jira and continue
>>>> to work on it.
>>>> --
>>>> thanks
>>>> ashish
>>>> Blog: http://www.ashishpaliwal.com/blog
>>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
>> --
>> thanks
>> ashish
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Hari Shreedharan <hs...@cloudera.com>.
I think it is best to replace the current system rather than having multiple metrics systems. We should implement it such that it is transparent to user code, and the changes only go into MonitoredCounterGroup (the CounterGroup class is obsolete - dont bother).
Thanks,
Hari
On Mon, Nov 10, 2014 at 5:36 PM, Ashish <pa...@gmail.com> wrote:
> Codahale does have lot more features like the rate/sec one, which is
> most needed metrics. It provides a lot of reporters out of box
> (Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
> enable it, rather than writing custom components.
> As of now we have to hide it within MonitoredCounterGroup and its
> sub-classes, that's what I did for running it. It was running parallel
> to existing system. Let me work more on it and see if I can simplify
> the same.
> On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
> <hs...@cloudera.com> wrote:
>> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>>
>>
>> Thanks,
>> Hari
>>
>> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>>
>>> Hi,
>>> Have hacked a bit into our existing instrumentation package and piggy
>>> backed cohahale metrics package. Here is one sample for Spooled
>>> Directory source (with instrumentation only for Source and Channel ),
>>> using console reporter
>>> -- Gauges ----------------------------------------------------------------------
>>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>>> value = 200
>>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>>> value = 2.0
>>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>>> value = 0
>>> -- Counters --------------------------------------------------------------------
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>>> count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>>> count = 1138800
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>>> count = 1138601
>>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>>> count = 1138600
>>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>>> count = 1138800
>>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>>> count = 1138800
>>> src.append-batch.accepted
>>> count = 11388
>>> src.append-batch.received
>>> count = 11388
>>> src.append.accepted
>>> count = 0
>>> src.append.received
>>> count = 0
>>> -- Meters ----------------------------------------------------------------------
>>> eventAcceptedMeter
>>> count = 1138800
>>> mean rate = 106478.56 events/second
>>> 1-minute rate = 93420.18 events/second
>>> 5-minute rate = 91389.24 events/second
>>> 15-minute rate = 91037.40 events/second
>>> eventReceivedMeter
>>> count = 1138800
>>> mean rate = 106462.14 events/second
>>> 1-minute rate = 93420.18 events/second
>>> 5-minute rate = 91389.24 events/second
>>> 15-minute rate = 91037.40 events/second
>>> If there is interest in the community, can raise a jira and continue
>>> to work on it.
>>> --
>>> thanks
>>> ashish
>>> Blog: http://www.ashishpaliwal.com/blog
>>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Ashish <pa...@gmail.com>.
Codahale does have lot more features like the rate/sec one, which is
most needed metrics. It provides a lot of reporters out of box
(Nagios, HTTP, Ganglia, Graphite, Console etc) so we just need to
enable it, rather than writing custom components.
As of now we have to hide it within MonitoredCounterGroup and its
sub-classes, that's what I did for running it. It was running parallel
to existing system. Let me work more on it and see if I can simplify
the same.
On Tue, Nov 11, 2014 at 4:50 AM, Hari Shreedharan
<hs...@cloudera.com> wrote:
> Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
>
>
> Thanks,
> Hari
>
> On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
>
>> Hi,
>> Have hacked a bit into our existing instrumentation package and piggy
>> backed cohahale metrics package. Here is one sample for Spooled
>> Directory source (with instrumentation only for Source and Channel ),
>> using console reporter
>> -- Gauges ----------------------------------------------------------------------
>> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
>> value = 200
>> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
>> value = 2.0
>> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
>> value = 0
>> -- Counters --------------------------------------------------------------------
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
>> count = 1138800
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
>> count = 1138800
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
>> count = 1138601
>> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
>> count = 1138600
>> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
>> count = 1138800
>> org.apache.flume.instrumentation.SourceCounter.src.events.received
>> count = 1138800
>> src.append-batch.accepted
>> count = 11388
>> src.append-batch.received
>> count = 11388
>> src.append.accepted
>> count = 0
>> src.append.received
>> count = 0
>> -- Meters ----------------------------------------------------------------------
>> eventAcceptedMeter
>> count = 1138800
>> mean rate = 106478.56 events/second
>> 1-minute rate = 93420.18 events/second
>> 5-minute rate = 91389.24 events/second
>> 15-minute rate = 91037.40 events/second
>> eventReceivedMeter
>> count = 1138800
>> mean rate = 106462.14 events/second
>> 1-minute rate = 93420.18 events/second
>> 5-minute rate = 91389.24 events/second
>> 15-minute rate = 91037.40 events/second
>> If there is interest in the community, can raise a jira and continue
>> to work on it.
>> --
>> thanks
>> ashish
>> Blog: http://www.ashishpaliwal.com/blog
>> My Photo Galleries: http://www.pbase.com/ashishpaliwal
--
thanks
ashish
Blog: http://www.ashishpaliwal.com/blog
My Photo Galleries: http://www.pbase.com/ashishpaliwal
Re: Using CodaHale metrics for monitoring
Posted by Hari Shreedharan <hs...@cloudera.com>.
Is it easier to use than the current one and/or does it give better performance? You’d need to support the current metrics API (MonitoredCounterGroup, SourceCounter, SinkCounter, ChannelCounter etc).
Thanks,
Hari
On Sat, Nov 8, 2014 at 8:21 PM, Ashish <pa...@gmail.com> wrote:
> Hi,
> Have hacked a bit into our existing instrumentation package and piggy
> backed cohahale metrics package. Here is one sample for Spooled
> Directory source (with instrumentation only for Source and Channel ),
> using console reporter
> -- Gauges ----------------------------------------------------------------------
> org.apache.flume.instrumentation.ChannelCounter.channel.current.size
> value = 200
> org.apache.flume.instrumentation.ChannelCounter.channel.fill.percentage
> value = 2.0
> org.apache.flume.instrumentation.SourceCounter.src.open-connection.count
> value = 0
> -- Counters --------------------------------------------------------------------
> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.attempt
> count = 1138800
> org.apache.flume.instrumentation.ChannelCounter.channel.event.put.success
> count = 1138800
> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.attempt
> count = 1138601
> org.apache.flume.instrumentation.ChannelCounter.channel.event.take.success
> count = 1138600
> org.apache.flume.instrumentation.SourceCounter.src.events.accepted
> count = 1138800
> org.apache.flume.instrumentation.SourceCounter.src.events.received
> count = 1138800
> src.append-batch.accepted
> count = 11388
> src.append-batch.received
> count = 11388
> src.append.accepted
> count = 0
> src.append.received
> count = 0
> -- Meters ----------------------------------------------------------------------
> eventAcceptedMeter
> count = 1138800
> mean rate = 106478.56 events/second
> 1-minute rate = 93420.18 events/second
> 5-minute rate = 91389.24 events/second
> 15-minute rate = 91037.40 events/second
> eventReceivedMeter
> count = 1138800
> mean rate = 106462.14 events/second
> 1-minute rate = 93420.18 events/second
> 5-minute rate = 91389.24 events/second
> 15-minute rate = 91037.40 events/second
> If there is interest in the community, can raise a jira and continue
> to work on it.
> --
> thanks
> ashish
> Blog: http://www.ashishpaliwal.com/blog
> My Photo Galleries: http://www.pbase.com/ashishpaliwal