You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Anis Nasir <aa...@gmail.com> on 2014/06/16 22:37:41 UTC
Extracting Performance Metrics
Dear all,
I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
zookeeper nodes. I am executing multiple topologies on the cluster and I
want to extract different metrics that I am mentioning below. Can someone
help me by recommending tools that I can use to extract this information.
Per Topology
- Throughput
- Latency
Per Spout or Bolt
- Throughput
- Latency
- Execution Time
- Queuing Time
- Number of Messages Processed
Regards
Anis
Re: Extracting Performance Metrics
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Anis,
I'm a little late to this thread, but have a look at SPM for Storm:
http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
This may also be of interest to those using Kafka to feed Storm or those
who use Storm with Cassandra:
http://blog.sematext.com/2014/06/11/monitoring-kafka-storm-and-cassandra-together-with-spm/
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
> Dear all,
>
> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
> zookeeper nodes. I am executing multiple topologies on the cluster and I
> want to extract different metrics that I am mentioning below. Can someone
> help me by recommending tools that I can use to extract this information.
>
>
> Per Topology
> - Throughput
> - Latency
>
> Per Spout or Bolt
> - Throughput
> - Latency
> - Execution Time
> - Queuing Time
> - Number of Messages Processed
>
> Regards
> Anis
>
Re: Extracting Performance Metrics
Posted by Ted Dunning <te...@gmail.com>.
If you can afford a bit more time for insertion, consider also t-digest.
Differences relative to the high dynamic range histogram system include:
- HDR histograms assume an exponential distribution. t-digest handles
arbitrary distributions
- t-digest is much more accurate near extreme values. You can have 0.1%
accuracy near the 50th percentile and have part per million accuracy (or
better) at the 99.999th %-ile
- HDR histograms are *much* faster for insertion. The claimed speeds are <
10ns for HDF histograms and measured speeds for t-digest are more like
200-500 ns.
- t-digest can handle far more skew than HDR histograms. A standard test
case is a gamma distribution which has 30 orders of magnitude of skew near
low quantiles.
See https://github.com/tdunning/t-digest for a low transitive dependency
version. Apache Mahout also has a version (with larger dependency tree).
Let me know if I can help on this.
On Mon, Jun 16, 2014 at 5:57 PM, Dan <dc...@hotmail.com> wrote:
> Be careful when using Coda Hale's Metrics package when measuring latency.
> Consider using Gil Tene's
> High Dynamic Range Histogram instead:
>
> http://hdrhistogram.github.io/HdrHistogram/
>
> -Dan
>
> ------------------------------
> From: andrew@parsely.com
> Date: Mon, 16 Jun 2014 18:20:11 -0400
> Subject: Re: Extracting Performance Metrics
> To: user@storm.incubator.apache.org
>
> Also, I came across this presentation by Visible Measures which actually
> walks through a lot of great options covering most of what you want to know
> about:
>
> http://files.meetup.com/5809742/storm%20monitoring.pdf
>
> One other thing to be aware of is that in Storm 0.9.2 (forthcoming
> release), there is a new REST API used by the Storm UI for gathering some
> of these metrics:
>
> https://github.com/apache/incubator-storm/pull/101
> https://issues.apache.org/jira/browse/STORM-205
>
>
> On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com>
> wrote:
>
> I haven't used it yet, but a lot of people get pointed to metrics_storm:
>
> https://github.com/ooyala/metrics_storm
>
> With this blog post that discusses it:
>
> http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
>
> Michael Noll also has a nice blog post about streaming Storm 0.9 metrics
> to Graphite:
>
>
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> Currently, when we use Storm, we do a lot of custom metrics in Graphite
> using statsd, as described in this post (not about Storm, but about
> Graphite/statsd):
>
> http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
>
>
>
>
> On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
>
> Dear all,
>
> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
> zookeeper nodes. I am executing multiple topologies on the cluster and I
> want to extract different metrics that I am mentioning below. Can someone
> help me by recommending tools that I can use to extract this information.
>
>
> Per Topology
> - Throughput
> - Latency
>
> Per Spout or Bolt
> - Throughput
> - Latency
> - Execution Time
> - Queuing Time
> - Number of Messages Processed
>
> Regards
> Anis
>
>
>
>
RE: Extracting Performance Metrics
Posted by Dan <dc...@hotmail.com>.
This post from Gil Tene sums up the Metrics package's issues with measuring latency well:
https://groups.google.com/forum/#!msg/mechanical-sympathy/I4JfZQ1GYi8/LKc3MYXRGWUJ
Date: Mon, 16 Jun 2014 19:00:30 -0600
Subject: RE: Extracting Performance Metrics
From: michael@fullcontact.com
To: user@storm.incubator.apache.org
What kind of issues does Metrics have that leads you to recommend HdrHistogram?
On Jun 16, 2014 6:57 PM, "Dan" <dc...@hotmail.com> wrote:
Be careful when using Coda Hale's Metrics package when measuring latency. Consider using Gil Tene'sHigh Dynamic Range Histogram instead:
http://hdrhistogram.github.io/HdrHistogram/
-Dan
From: andrew@parsely.com
Date: Mon, 16 Jun 2014 18:20:11 -0400
Subject: Re: Extracting Performance Metrics
To: user@storm.incubator.apache.org
Also, I came across this presentation by Visible Measures which actually walks through a lot of great options covering most of what you want to know about:
http://files.meetup.com/5809742/storm%20monitoring.pdf
One other thing to be aware of is that in Storm 0.9.2 (forthcoming release), there is a new REST API used by the Storm UI for gathering some of these metrics:
https://github.com/apache/incubator-storm/pull/101
https://issues.apache.org/jira/browse/STORM-205
On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com> wrote:
I haven't used it yet, but a lot of people get pointed to metrics_storm:
https://github.com/ooyala/metrics_storm
With this blog post that discusses it:
http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to Graphite:
http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
Currently, when we use Storm, we do a lot of custom metrics in Graphite using statsd, as described in this post (not about Storm, but about Graphite/statsd):
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
Dear all,
I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1 zookeeper nodes. I am executing multiple topologies on the cluster and I want to extract different metrics that I am mentioning below. Can someone help me by recommending tools that I can use to extract this information.
Per Topology - Throughput - Latency
Per Spout or Bolt - Throughput - Latency - Execution Time
- Queuing Time - Number of Messages Processed
RegardsAnis
Re: Extracting Performance Metrics
Posted by Ted Dunning <te...@gmail.com>.
CodeHale doesn't handle extreme skew on measurements well last time I
looked. For throughput, averages are great. For latency, you need very
high percentiles to understand what is happening.
On Mon, Jun 16, 2014 at 6:00 PM, Michael Rose <mi...@fullcontact.com>
wrote:
> What kind of issues does Metrics have that leads you to recommend
> HdrHistogram?
> On Jun 16, 2014 6:57 PM, "Dan" <dc...@hotmail.com> wrote:
>
>> Be careful when using Coda Hale's Metrics package when measuring latency.
>> Consider using Gil Tene's
>> High Dynamic Range Histogram instead:
>>
>> http://hdrhistogram.github.io/HdrHistogram/
>>
>> -Dan
>>
>> ------------------------------
>> From: andrew@parsely.com
>> Date: Mon, 16 Jun 2014 18:20:11 -0400
>> Subject: Re: Extracting Performance Metrics
>> To: user@storm.incubator.apache.org
>>
>> Also, I came across this presentation by Visible Measures which actually
>> walks through a lot of great options covering most of what you want to know
>> about:
>>
>> http://files.meetup.com/5809742/storm%20monitoring.pdf
>>
>> One other thing to be aware of is that in Storm 0.9.2 (forthcoming
>> release), there is a new REST API used by the Storm UI for gathering some
>> of these metrics:
>>
>> https://github.com/apache/incubator-storm/pull/101
>> https://issues.apache.org/jira/browse/STORM-205
>>
>>
>> On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com>
>> wrote:
>>
>> I haven't used it yet, but a lot of people get pointed to metrics_storm:
>>
>> https://github.com/ooyala/metrics_storm
>>
>> With this blog post that discusses it:
>>
>> http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
>>
>> Michael Noll also has a nice blog post about streaming Storm 0.9 metrics
>> to Graphite:
>>
>>
>> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>>
>> Currently, when we use Storm, we do a lot of custom metrics in Graphite
>> using statsd, as described in this post (not about Storm, but about
>> Graphite/statsd):
>>
>> http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
>>
>>
>>
>>
>> On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
>>
>> Dear all,
>>
>> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
>> zookeeper nodes. I am executing multiple topologies on the cluster and I
>> want to extract different metrics that I am mentioning below. Can someone
>> help me by recommending tools that I can use to extract this information.
>>
>>
>> Per Topology
>> - Throughput
>> - Latency
>>
>> Per Spout or Bolt
>> - Throughput
>> - Latency
>> - Execution Time
>> - Queuing Time
>> - Number of Messages Processed
>>
>> Regards
>> Anis
>>
>>
>>
>>
RE: Extracting Performance Metrics
Posted by Michael Rose <mi...@fullcontact.com>.
What kind of issues does Metrics have that leads you to recommend
HdrHistogram?
On Jun 16, 2014 6:57 PM, "Dan" <dc...@hotmail.com> wrote:
> Be careful when using Coda Hale's Metrics package when measuring latency.
> Consider using Gil Tene's
> High Dynamic Range Histogram instead:
>
> http://hdrhistogram.github.io/HdrHistogram/
>
> -Dan
>
> ------------------------------
> From: andrew@parsely.com
> Date: Mon, 16 Jun 2014 18:20:11 -0400
> Subject: Re: Extracting Performance Metrics
> To: user@storm.incubator.apache.org
>
> Also, I came across this presentation by Visible Measures which actually
> walks through a lot of great options covering most of what you want to know
> about:
>
> http://files.meetup.com/5809742/storm%20monitoring.pdf
>
> One other thing to be aware of is that in Storm 0.9.2 (forthcoming
> release), there is a new REST API used by the Storm UI for gathering some
> of these metrics:
>
> https://github.com/apache/incubator-storm/pull/101
> https://issues.apache.org/jira/browse/STORM-205
>
>
> On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com>
> wrote:
>
> I haven't used it yet, but a lot of people get pointed to metrics_storm:
>
> https://github.com/ooyala/metrics_storm
>
> With this blog post that discusses it:
>
> http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
>
> Michael Noll also has a nice blog post about streaming Storm 0.9 metrics
> to Graphite:
>
>
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> Currently, when we use Storm, we do a lot of custom metrics in Graphite
> using statsd, as described in this post (not about Storm, but about
> Graphite/statsd):
>
> http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
>
>
>
>
> On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
>
> Dear all,
>
> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
> zookeeper nodes. I am executing multiple topologies on the cluster and I
> want to extract different metrics that I am mentioning below. Can someone
> help me by recommending tools that I can use to extract this information.
>
>
> Per Topology
> - Throughput
> - Latency
>
> Per Spout or Bolt
> - Throughput
> - Latency
> - Execution Time
> - Queuing Time
> - Number of Messages Processed
>
> Regards
> Anis
>
>
>
>
RE: Extracting Performance Metrics
Posted by Dan <dc...@hotmail.com>.
Be careful when using Coda Hale's Metrics package when measuring latency. Consider using Gil Tene'sHigh Dynamic Range Histogram instead:
http://hdrhistogram.github.io/HdrHistogram/
-Dan
From: andrew@parsely.com
Date: Mon, 16 Jun 2014 18:20:11 -0400
Subject: Re: Extracting Performance Metrics
To: user@storm.incubator.apache.org
Also, I came across this presentation by Visible Measures which actually walks through a lot of great options covering most of what you want to know about:
http://files.meetup.com/5809742/storm%20monitoring.pdf
One other thing to be aware of is that in Storm 0.9.2 (forthcoming release), there is a new REST API used by the Storm UI for gathering some of these metrics:
https://github.com/apache/incubator-storm/pull/101
https://issues.apache.org/jira/browse/STORM-205
On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com> wrote:
I haven't used it yet, but a lot of people get pointed to metrics_storm:
https://github.com/ooyala/metrics_storm
With this blog post that discusses it:
http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to Graphite:
http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
Currently, when we use Storm, we do a lot of custom metrics in Graphite using statsd, as described in this post (not about Storm, but about Graphite/statsd):
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
Dear all,
I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1 zookeeper nodes. I am executing multiple topologies on the cluster and I want to extract different metrics that I am mentioning below. Can someone help me by recommending tools that I can use to extract this information.
Per Topology - Throughput - Latency
Per Spout or Bolt - Throughput - Latency - Execution Time
- Queuing Time - Number of Messages Processed
RegardsAnis
Re: Extracting Performance Metrics
Posted by Andrew Montalenti <an...@parsely.com>.
Also, I came across this presentation by Visible Measures which actually
walks through a lot of great options covering most of what you want to know
about:
http://files.meetup.com/5809742/storm%20monitoring.pdf
One other thing to be aware of is that in Storm 0.9.2 (forthcoming
release), there is a new REST API used by the Storm UI for gathering some
of these metrics:
https://github.com/apache/incubator-storm/pull/101
https://issues.apache.org/jira/browse/STORM-205
On Mon, Jun 16, 2014 at 6:13 PM, Andrew Montalenti <an...@parsely.com>
wrote:
> I haven't used it yet, but a lot of people get pointed to metrics_storm:
>
> https://github.com/ooyala/metrics_storm
>
> With this blog post that discusses it:
>
> http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
>
> Michael Noll also has a nice blog post about streaming Storm 0.9 metrics
> to Graphite:
>
>
> http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
>
> Currently, when we use Storm, we do a lot of custom metrics in Graphite
> using statsd, as described in this post (not about Storm, but about
> Graphite/statsd):
>
> http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
>
>
>
>
> On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
>
>> Dear all,
>>
>> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
>> zookeeper nodes. I am executing multiple topologies on the cluster and I
>> want to extract different metrics that I am mentioning below. Can someone
>> help me by recommending tools that I can use to extract this information.
>>
>>
>> Per Topology
>> - Throughput
>> - Latency
>>
>> Per Spout or Bolt
>> - Throughput
>> - Latency
>> - Execution Time
>> - Queuing Time
>> - Number of Messages Processed
>>
>> Regards
>> Anis
>>
>
>
Re: Extracting Performance Metrics
Posted by Andrew Montalenti <an...@parsely.com>.
I haven't used it yet, but a lot of people get pointed to metrics_storm:
https://github.com/ooyala/metrics_storm
With this blog post that discusses it:
http://engineering.ooyala.com/blog/open-sourcing-metrics-storm
Michael Noll also has a nice blog post about streaming Storm 0.9 metrics to
Graphite:
http://www.michael-noll.com/blog/2013/11/06/sending-metrics-from-storm-to-graphite/
Currently, when we use Storm, we do a lot of custom metrics in Graphite
using statsd, as described in this post (not about Storm, but about
Graphite/statsd):
http://codeascraft.com/2011/02/15/measure-anything-measure-everything/
On Mon, Jun 16, 2014 at 4:37 PM, Anis Nasir <aa...@gmail.com> wrote:
> Dear all,
>
> I am running a cluster with 1 kafka + 1 nimbus + 10 supervisor + 1
> zookeeper nodes. I am executing multiple topologies on the cluster and I
> want to extract different metrics that I am mentioning below. Can someone
> help me by recommending tools that I can use to extract this information.
>
>
> Per Topology
> - Throughput
> - Latency
>
> Per Spout or Bolt
> - Throughput
> - Latency
> - Execution Time
> - Queuing Time
> - Number of Messages Processed
>
> Regards
> Anis
>