You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Chris Bedford <ch...@buildlackey.com> on 2014/03/18 01:21:53 UTC
Best way to get JMX access to storm metrics for spout tuples emitted
/ acked | bolt tuples emitted, latency etc.
Hi, I'm interested in getting metrics via JMX on not only " container
level " factors (such as # of garbage collects, heap usage,etc.), but
also metrics that describe how spouts and bolts are performing (e.g., # of
tuples emitted, # transferred -- the same kind of stuff that the storm UI
shows.)
I ran across this project :
:
https://github.com/ooyala/metrics_storm/blob/master/src/ooyala/common/metrics_storm/MetricsStorm.scala
I was just wondering if this is the best best to pursue. Does anyone have
any concrete experience with this that they can share?
thanx!
chris
Re: Best way to get JMX access to storm metrics for spout tuples
emitted / acked | bolt tuples emitted, latency etc.
Posted by Chris Bedford <ch...@buildlackey.com>.
Thanks, Noel and Michael !
On Mon, Mar 17, 2014 at 5:55 PM, Noel Milton Vega <
nmvega@computingarchitects.com> wrote:
> I used Nimbus Thrift on my end.
>
> The one thing to note about that approach is that, if you're planning to
> plot
> those metrics (say, on Graphite), the results aren't very interesting
> because the
> metrics -- emitted, transferred, acks, etc. -- are aggregates and simply
> monotonically
> increase, which lead to not so insightful graphs (... more-or-less lines
> that
> go up from left-to-right forever).
>
> However, I took consecutive metrics samples from Nimbus, say,
> every 5 seconds, and plotted the difference between the two samples
> divided by 5 to plot
> average quantities instead. That was insightful. So while (except for
> latency
> type metrics) Nimbus doesn't provide you with average rates for
> emits/transferred/acks
> (and you won't see them on the UI either), you can easily derive them
> manually
> and send that to Graphite (or to your charting platform). If using
> Graphite, btw.,
> you also have the option of letting Graphite calculate the averages on the
> back-end
> for you, too (giving you options).
>
> Finally, for application specific metrics exposed via Java Mbeans,
> consider Codahale's
> Metric's API (in-band), and JMXtrans (out-of-band).
>
> Noel Milton Vega
> Dimension Data, LLC.
> nmvega@didata.us
> nmvega@ComputingArchitects.Com
>
>
>
> On 03/17/2014 08:30 PM, Michael Rose wrote:
>
> Depending on what you need, you could use the Thrift interface to Nimbus
> to grab the statistics. The UI project does do some calculations on the raw
> metrics, so it's not quite the same. The algorithms it uses aren't too
> difficult to replicate.
>
> We ended up building something very similar to what Ooyala did,
> customized for our specific needs, it's an excellent pattern.
>
> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
> michael@fullcontact.com
>
>
> On Mon, Mar 17, 2014 at 6:21 PM, Chris Bedford <ch...@buildlackey.com>wrote:
>
>>
>>
>> Hi, I'm interested in getting metrics via JMX on not only " container
>> level " factors (such as # of garbage collects, heap usage,etc.), but
>> also metrics that describe how spouts and bolts are performing (e.g., # of
>> tuples emitted, # transferred -- the same kind of stuff that the storm UI
>> shows.)
>>
>> I ran across this project :
>> :
>> https://github.com/ooyala/metrics_storm/blob/master/src/ooyala/common/metrics_storm/MetricsStorm.scala
>>
>>
>> I was just wondering if this is the best best to pursue. Does anyone
>> have any concrete experience with this that they can share?
>> thanx!
>>
>> chris
>>
>>
>
>
--
Chris Bedford
Founder & Lead Lackey
Build Lackey Labs: http://buildlackey.com
Go Grails!: http://blog.buildlackey.com
Re: Best way to get JMX access to storm metrics for spout tuples
emitted / acked | bolt tuples emitted, latency etc.
Posted by Noel Milton Vega <nm...@ComputingArchitects.Com>.
I used Nimbus Thrift on my end.
The one thing to note about that approach is that, if you're planning to
plot
those metrics (say, on Graphite), the results aren't very interesting
because the
metrics -- emitted, transferred, acks, etc. -- are aggregates and simply
monotonically
increase, which lead to not so insightful graphs (... more-or-less lines
that
go up from left-to-right forever).
However, I took consecutive metrics samples from Nimbus, say,
every 5 seconds, and plotted the difference between the two samples
divided by 5 to plot
average quantities instead. That was insightful. So while (except for
latency
type metrics) Nimbus doesn't provide you with average rates for
emits/transferred/acks
(and you won't see them on the UI either), you can easily derive them
manually
and send that to Graphite (or to your charting platform). If using
Graphite, btw.,
you also have the option of letting Graphite calculate the averages on
the back-end
for you, too (giving you options).
Finally, for application specific metrics exposed via Java Mbeans,
consider Codahale's
Metric's API (in-band), and JMXtrans (out-of-band).
Noel Milton Vega
Dimension Data, LLC.
nmvega@didata.us
nmvega@ComputingArchitects.Com
On 03/17/2014 08:30 PM, Michael Rose wrote:
> Depending on what you need, you could use the Thrift interface to
> Nimbus to grab the statistics. The UI project does do some
> calculations on the raw metrics, so it's not quite the same. The
> algorithms it uses aren't too difficult to replicate.
>
> We ended up building something very similar to what Ooyala did,
> customized for our specific needs, it's an excellent pattern.
>
> Michael Rose (@Xorlev <https://twitter.com/xorlev>)
> Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
> michael@fullcontact.com <ma...@fullcontact.com>
>
>
>
> On Mon, Mar 17, 2014 at 6:21 PM, Chris Bedford <chris@buildlackey.com
> <ma...@buildlackey.com>> wrote:
>
>
> Hi, I'm interested in getting metrics via JMX on not only "
> container level " factors (such as # of garbage collects, heap
> usage,etc.), but also metrics that describe how spouts and bolts
> are performing (e.g., # of tuples emitted, # transferred -- the
> same kind of stuff that the storm UI
> shows.)
>
> I ran across this project :
> :
> https://github.com/ooyala/metrics_storm/blob/master/src/ooyala/common/metrics_storm/MetricsStorm.scala
>
>
> I was just wondering if this is the best best to pursue. Does
> anyone have any concrete experience with this that they can share?
> thanx!
>
> chris
>
>
Re: Best way to get JMX access to storm metrics for spout tuples
emitted / acked | bolt tuples emitted, latency etc.
Posted by Michael Rose <mi...@fullcontact.com>.
Depending on what you need, you could use the Thrift interface to Nimbus to
grab the statistics. The UI project does do some calculations on the raw
metrics, so it's not quite the same. The algorithms it uses aren't too
difficult to replicate.
We ended up building something very similar to what Ooyala did, customized
for our specific needs, it's an excellent pattern.
Michael Rose (@Xorlev <https://twitter.com/xorlev>)
Senior Platform Engineer, FullContact <http://www.fullcontact.com/>
michael@fullcontact.com
On Mon, Mar 17, 2014 at 6:21 PM, Chris Bedford <ch...@buildlackey.com>wrote:
>
>
> Hi, I'm interested in getting metrics via JMX on not only " container
> level " factors (such as # of garbage collects, heap usage,etc.), but
> also metrics that describe how spouts and bolts are performing (e.g., # of
> tuples emitted, # transferred -- the same kind of stuff that the storm UI
> shows.)
>
> I ran across this project :
> :
> https://github.com/ooyala/metrics_storm/blob/master/src/ooyala/common/metrics_storm/MetricsStorm.scala
>
>
> I was just wondering if this is the best best to pursue. Does anyone have
> any concrete experience with this that they can share?
> thanx!
>
> chris
>
>