You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Raphael Hsieh <ra...@gmail.com> on 2014/09/20 00:12:52 UTC
metrics consumer logging stormUI data
Hi,
Using Storm/Trident, how do I register a metrics consumer to log the data I
get in the StormUI ?
I want to look at historical data of my topology, for example the execute
latency of the topology over time, as this would give me good insight as to
where things might be going wrong when the system breaks.
I have been following the steps outlined in the BigData CookBook here:
http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
However I am not wanting to create my own metrics, instead I just want to
log the metrics that already exist built in to Storm. It is unclear to me
how I am supposed to go about doing that.
Thanks
--
Raphael Hsieh
Re: metrics consumer logging stormUI data
Posted by Harsha <st...@harsha.io>.
Hi Raphael,
I tested it with wordcounttopology under examples.
conf.registerMetricsConsumer(LoggingMetricsConsumer.class,
2);
I do see the metrics added to the logs/metrics.log. metrics.log
should be present by default under storm/logs dir.
-Harsha
On Mon, Sep 22, 2014, at 09:24 AM, Raphael Hsieh wrote:
Thanks Harsha and Otis for your prompt responses.
I'm looking to somehow log these metrics to use for an in-house
monitoring system. I don't want to get user provided metrics
just yet.
>From what I've gathered from the big data cookbook is that I
just want to create a metrics consumer to read these metrics
and print it out to a log file. In order to do this I have
added to my config:
config.registerMetricsConsumer(LoggingMetricsConsumer.class,
2);
which should create a loggingMetricsConsumer with a parallelism
of 2 (I believe). I was lead to believe that these logs would
be put in a file called "metrics.log". However after adding
this to my topology I have been unable to find such a log.
If someone could explain to me what I might be missing that
would be great.
Thanks!
On Fri, Sep 19, 2014 at 4:27 PM, Gezim Musliaj
<[1...@gmail.com> wrote:
Hey Otis, I was just registered at sematext and I can say that
this is what I have been looking for.I have just one question,
what about the delays between the SPM and the Storm Cluster (if
they do exist), whats the worst case? I mean because these
metrics are not calculated locally, but using an internet
connection.
Thanks !
On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic
<[2...@gmail.com> wrote:
Raphael,
Not sure if this is what you are after, but [3]SPM will collect
and graph all Storm metrics, let you do alerting and anomaly
detection on them, etc. If you want to graph custom metrics
(e.g. something from your bolts), you can send them in as
[4]custom metrics and again graph them, alert on them, do
anomaly detection on them, stick them on dashboards, etc. If
you want to emit events from your bolts, you can [5]send events
to SPM, too, or you can send them to [6]Logsene... can be handy
for correlation with alerts and performance graphs when
troubleshooting. Here are some Storm metrics graph:
[7]http://blog.sematext.com/2014/01/30/announcement-apache-stor
m-monitoring-in-spm/
I hope this helps.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
Management
Solr & Elasticsearch Support * [8]http://sematext.com/
On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh
<[9...@gmail.com> wrote:
Hi,
Using Storm/Trident, how do I register a metrics consumer to
log the data I get in the StormUI ?
I want to look at historical data of my topology, for example
the execute latency of the topology over time, as this would
give me good insight as to where things might be going wrong
when the system breaks.
I have been following the steps outlined in the BigData
CookBook
here: [10]http://www.bigdata-cookbook.com/post/72320512609/stor
m-metrics-how-to
However I am not wanting to create my own metrics, instead I
just want to log the metrics that already exist built in to
Storm. It is unclear to me how I am supposed to go about doing
that.
Thanks
--
Raphael Hsieh
--
Raphael Hsieh
References
1. mailto:gmusliaj@gmail.com
2. mailto:otis.gospodnetic@gmail.com
3. http://sematext.com/spm/
4. https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics
5. https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration
6. http://www.sematext.com/logsene/
7. http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
8. http://sematext.com/
9. mailto:raffihsieh@gmail.com
10. http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Re: metrics consumer logging stormUI data
Posted by Raphael Hsieh <ra...@gmail.com>.
Thanks Harsha and Otis for your prompt responses.
I'm looking to somehow log these metrics to use for an in-house monitoring
system. I don't want to get user provided metrics just yet.
>From what I've gathered from the big data cookbook is that I just want to
create a metrics consumer to read these metrics and print it out to a log
file. In order to do this I have added to my config:
config.registerMetricsConsumer(LoggingMetricsConsumer.class, 2);
which should create a loggingMetricsConsumer with a parallelism of 2 (I
believe). I was lead to believe that these logs would be put in a file
called "metrics.log". However after adding this to my topology I have been
unable to find such a log.
If someone could explain to me what I might be missing that would be great.
Thanks!
On Fri, Sep 19, 2014 at 4:27 PM, Gezim Musliaj <gm...@gmail.com> wrote:
> Hey Otis, I was just registered at sematext and I can say that this is
> what I have been looking for.I have just one question, what about the
> delays between the SPM and the Storm Cluster (if they do exist), whats the
> worst case? I mean because these metrics are not calculated locally, but
> using an internet connection.
> Thanks !
>
> On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Raphael,
>>
>> Not sure if this is what you are after, but SPM
>> <http://sematext.com/spm/> will collect and graph all Storm metrics, let
>> you do alerting and anomaly detection on them, etc. If you want to graph
>> custom metrics (e.g. something from your bolts), you can send them in as custom
>> metrics
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
>> again graph them, alert on them, do anomaly detection on them, stick them
>> on dashboards, etc. If you want to emit events from your bolts, you can send
>> events to SPM
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
>> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
>> can be handy for correlation with alerts and performance graphs when
>> troubleshooting. Here are some Storm metrics graph:
>> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>>
>>
>> I hope this helps.
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Using Storm/Trident, how do I register a metrics consumer to log the
>>> data I get in the StormUI ?
>>> I want to look at historical data of my topology, for example the
>>> execute latency of the topology over time, as this would give me good
>>> insight as to where things might be going wrong when the system breaks.
>>>
>>> I have been following the steps outlined in the BigData CookBook here:
>>> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>>>
>>> However I am not wanting to create my own metrics, instead I just want
>>> to log the metrics that already exist built in to Storm. It is unclear to
>>> me how I am supposed to go about doing that.
>>>
>>> Thanks
>>>
>>> --
>>> Raphael Hsieh
>>>
>>>
>>>
>>>
>>
>>
>
--
Raphael Hsieh
Re: metrics consumer logging stormUI data
Posted by John Reilly <jr...@inconspicuous.org>.
It is not necessary to have debug enabled to get metrics. Also, it is
independent of storm ui. The ui does not need to be running - the metrics
are reported by the workers.
On Wed, Sep 24, 2014 at 12:12 PM, Raphael Hsieh <ra...@gmail.com>
wrote:
> In order to get this / have the metrics consumer work, do I need to have
> the setDebug attribute set to true?
>
> On Mon, Sep 22, 2014 at 12:59 PM, Harsha <st...@harsha.io> wrote:
>
>> Here is what I see in the metrics.log
>>
>> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
>> 19:split __transfer-count {default=2680}
>>
>> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
>> 19:split __execute-latency {spout:default=0.0}
>>
>> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
>> 19:split __fail-count {}
>>
>> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
>> 19:split __emit-count {default=2680}
>>
>> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
>> 19:split __execute-count {spout:default=420}
>>
>> 2014-09-22 09:44:31,352 73179 1411404271 localhost:6703
>> 22:split __ack-count {spout:default=420}
>>
>> 2014-09-22 09:44:31,352 73179 1411404271 localhost:6703
>> 22:split __sendqueue {write_pos=2679, capacity=1024,
>> read_pos=2679, population=0}
>> I do see all the UI related counts coming in the metrics.log.
>>
>> -Harsha
>>
>>
>> On Mon, Sep 22, 2014, at 10:41 AM, Raphael Hsieh wrote:
>>
>> Hi Harsha,
>> Did you have to bind the metrics consumer to the default StormUI metrics
>> at all? Or do those automagically get included ?
>>
>> Thanks!
>>
>> On Mon, Sep 22, 2014 at 10:33 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>> Hi Gezim,
>>
>> On Fri, Sep 19, 2014 at 7:27 PM, Gezim Musliaj <gm...@gmail.com>
>> wrote:
>>
>> Hey Otis, I was just registered at sematext and I can say that this is
>> what I have been looking for.I have just one question, what about the
>> delays between the SPM and the Storm Cluster (if they do exist), whats the
>> worst case? I mean because these metrics are not calculated locally, but
>> using an internet connection.
>>
>>
>>
>> The worst case is that somebody unplugs your servers from the network,
>> but if that happens you have bigger problems to deal with. In all
>> seriousness, Storm (local) => SPM (remote/cloud/saas) is not really a
>> problem -- lots of people successfully use SPM for monitoring Storm,
>> Hadoop, Kafka, and other types of systems.
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>>
>>
>> Thanks !
>>
>> On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>> Raphael,
>>
>> Not sure if this is what you are after, but SPM
>> <http://sematext.com/spm/> will collect and graph all Storm metrics, let
>> you do alerting and anomaly detection on them, etc. If you want to graph
>> custom metrics (e.g. something from your bolts), you can send them in as custom
>> metrics
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
>> again graph them, alert on them, do anomaly detection on them, stick them
>> on dashboards, etc. If you want to emit events from your bolts, you can send
>> events to SPM
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
>> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
>> can be handy for correlation with alerts and performance graphs when
>> troubleshooting. Here are some Storm metrics graph:
>> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>>
>> I hope this helps.
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
>> wrote:
>>
>> Hi,
>> Using Storm/Trident, how do I register a metrics consumer to log the data
>> I get in the StormUI ?
>> I want to look at historical data of my topology, for example the execute
>> latency of the topology over time, as this would give me good insight as to
>> where things might be going wrong when the system breaks.
>>
>> I have been following the steps outlined in the BigData CookBook here:
>> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>>
>> However I am not wanting to create my own metrics, instead I just want to
>> log the metrics that already exist built in to Storm. It is unclear to me
>> how I am supposed to go about doing that.
>>
>> Thanks
>>
>> --
>> Raphael Hsieh
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>>
>> --
>> Raphael Hsieh
>>
>>
>>
>>
>>
>>
>
>
>
> --
> Raphael Hsieh
>
>
>
>
Re: metrics consumer logging stormUI data
Posted by Raphael Hsieh <ra...@gmail.com>.
In order to get this / have the metrics consumer work, do I need to have
the setDebug attribute set to true?
On Mon, Sep 22, 2014 at 12:59 PM, Harsha <st...@harsha.io> wrote:
> Here is what I see in the metrics.log
>
> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
> 19:split __transfer-count {default=2680}
>
> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
> 19:split __execute-latency {spout:default=0.0}
>
> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
> 19:split __fail-count {}
>
> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
> 19:split __emit-count {default=2680}
>
> 2014-09-22 09:44:31,321 73175 1411404271 localhost:6703
> 19:split __execute-count {spout:default=420}
>
> 2014-09-22 09:44:31,352 73179 1411404271 localhost:6703
> 22:split __ack-count {spout:default=420}
>
> 2014-09-22 09:44:31,352 73179 1411404271 localhost:6703
> 22:split __sendqueue {write_pos=2679, capacity=1024,
> read_pos=2679, population=0}
> I do see all the UI related counts coming in the metrics.log.
>
> -Harsha
>
>
> On Mon, Sep 22, 2014, at 10:41 AM, Raphael Hsieh wrote:
>
> Hi Harsha,
> Did you have to bind the metrics consumer to the default StormUI metrics
> at all? Or do those automagically get included ?
>
> Thanks!
>
> On Mon, Sep 22, 2014 at 10:33 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> Hi Gezim,
>
> On Fri, Sep 19, 2014 at 7:27 PM, Gezim Musliaj <gm...@gmail.com> wrote:
>
> Hey Otis, I was just registered at sematext and I can say that this is
> what I have been looking for.I have just one question, what about the
> delays between the SPM and the Storm Cluster (if they do exist), whats the
> worst case? I mean because these metrics are not calculated locally, but
> using an internet connection.
>
>
>
> The worst case is that somebody unplugs your servers from the network, but
> if that happens you have bigger problems to deal with. In all seriousness,
> Storm (local) => SPM (remote/cloud/saas) is not really a problem -- lots of
> people successfully use SPM for monitoring Storm, Hadoop, Kafka, and other
> types of systems.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
>
> Thanks !
>
> On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
> Raphael,
>
> Not sure if this is what you are after, but SPM <http://sematext.com/spm/>
> will collect and graph all Storm metrics, let you do alerting and anomaly
> detection on them, etc. If you want to graph custom metrics (e.g.
> something from your bolts), you can send them in as custom metrics
> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
> again graph them, alert on them, do anomaly detection on them, stick them
> on dashboards, etc. If you want to emit events from your bolts, you can send
> events to SPM
> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
> can be handy for correlation with alerts and performance graphs when
> troubleshooting. Here are some Storm metrics graph:
> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>
> I hope this helps.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
> wrote:
>
> Hi,
> Using Storm/Trident, how do I register a metrics consumer to log the data
> I get in the StormUI ?
> I want to look at historical data of my topology, for example the execute
> latency of the topology over time, as this would give me good insight as to
> where things might be going wrong when the system breaks.
>
> I have been following the steps outlined in the BigData CookBook here:
> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>
> However I am not wanting to create my own metrics, instead I just want to
> log the metrics that already exist built in to Storm. It is unclear to me
> how I am supposed to go about doing that.
>
> Thanks
>
> --
> Raphael Hsieh
>
>
>
>
>
>
>
>
>
>
>
>
>
>
> --
> Raphael Hsieh
>
>
>
>
>
>
--
Raphael Hsieh
Re: metrics consumer logging stormUI data
Posted by Harsha <st...@harsha.io>.
Here is what I see in the metrics.log
2014-09-22 09:44:31,321 73175 1411404271
localhost:6703 19:split __transfer-count
{default=2680}
2014-09-22 09:44:31,321 73175 1411404271
localhost:6703 19:split __execute-latency
{spout:default=0.0}
2014-09-22 09:44:31,321 73175 1411404271
localhost:6703 19:split __fail-count {}
2014-09-22 09:44:31,321 73175 1411404271
localhost:6703 19:split __emit-count
{default=2680}
2014-09-22 09:44:31,321 73175 1411404271
localhost:6703 19:split __execute-count
{spout:default=420}
2014-09-22 09:44:31,352 73179 1411404271
localhost:6703 22:split __ack-count
{spout:default=420}
2014-09-22 09:44:31,352 73179 1411404271
localhost:6703 22:split __sendqueue
{write_pos=2679, capacity=1024, read_pos=2679, population=0}
I do see all the UI related counts coming in the metrics.log.
-Harsha
On Mon, Sep 22, 2014, at 10:41 AM, Raphael Hsieh wrote:
Hi Harsha,
Did you have to bind the metrics consumer to the default
StormUI metrics at all? Or do those automagically get included
?
Thanks!
On Mon, Sep 22, 2014 at 10:33 AM, Otis Gospodnetic
<[1...@gmail.com> wrote:
Hi Gezim,
On Fri, Sep 19, 2014 at 7:27 PM, Gezim Musliaj
<[2...@gmail.com> wrote:
Hey Otis, I was just registered at sematext and I can say that
this is what I have been looking for.I have just one question,
what about the delays between the SPM and the Storm Cluster (if
they do exist), whats the worst case? I mean because these
metrics are not calculated locally, but using an internet
connection.
The worst case is that somebody unplugs your servers from the
network, but if that happens you have bigger problems to deal
with. In all seriousness, Storm (local) => SPM
(remote/cloud/saas) is not really a problem -- lots of people
successfully use SPM for monitoring Storm, Hadoop, Kafka, and
other types of systems.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
Management
Solr & Elasticsearch Support * [3]http://sematext.com/
Thanks !
On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic
<[4...@gmail.com> wrote:
Raphael,
Not sure if this is what you are after, but [5]SPM will collect
and graph all Storm metrics, let you do alerting and anomaly
detection on them, etc. If you want to graph custom metrics
(e.g. something from your bolts), you can send them in as
[6]custom metrics and again graph them, alert on them, do
anomaly detection on them, stick them on dashboards, etc. If
you want to emit events from your bolts, you can [7]send events
to SPM, too, or you can send them to [8]Logsene... can be handy
for correlation with alerts and performance graphs when
troubleshooting. Here are some Storm metrics graph:
[9]http://blog.sematext.com/2014/01/30/announcement-apache-stor
m-monitoring-in-spm/
I hope this helps.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log
Management
Solr & Elasticsearch Support * [10]http://sematext.com/
On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh
<[1...@gmail.com> wrote:
Hi,
Using Storm/Trident, how do I register a metrics consumer to
log the data I get in the StormUI ?
I want to look at historical data of my topology, for example
the execute latency of the topology over time, as this would
give me good insight as to where things might be going wrong
when the system breaks.
I have been following the steps outlined in the BigData
CookBook
here: [12]http://www.bigdata-cookbook.com/post/72320512609/stor
m-metrics-how-to
However I am not wanting to create my own metrics, instead I
just want to log the metrics that already exist built in to
Storm. It is unclear to me how I am supposed to go about doing
that.
Thanks
--
Raphael Hsieh
--
Raphael Hsieh
References
1. mailto:otis.gospodnetic@gmail.com
2. mailto:gmusliaj@gmail.com
3. http://sematext.com/
4. mailto:otis.gospodnetic@gmail.com
5. http://sematext.com/spm/
6. https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics
7. https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration
8. http://www.sematext.com/logsene/
9. http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
10. http://sematext.com/
11. mailto:raffihsieh@gmail.com
12. http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
Re: metrics consumer logging stormUI data
Posted by Raphael Hsieh <ra...@gmail.com>.
Hi Harsha,
Did you have to bind the metrics consumer to the default StormUI metrics at
all? Or do those automagically get included ?
Thanks!
On Mon, Sep 22, 2014 at 10:33 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi Gezim,
>
> On Fri, Sep 19, 2014 at 7:27 PM, Gezim Musliaj <gm...@gmail.com> wrote:
>
>> Hey Otis, I was just registered at sematext and I can say that this is
>> what I have been looking for.I have just one question, what about the
>> delays between the SPM and the Storm Cluster (if they do exist), whats the
>> worst case? I mean because these metrics are not calculated locally, but
>> using an internet connection.
>>
>
> The worst case is that somebody unplugs your servers from the network, but
> if that happens you have bigger problems to deal with. In all seriousness,
> Storm (local) => SPM (remote/cloud/saas) is not really a problem -- lots of
> people successfully use SPM for monitoring Storm, Hadoop, Kafka, and other
> types of systems.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
>
>> Thanks !
>>
>> On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
>> otis.gospodnetic@gmail.com> wrote:
>>
>>> Raphael,
>>>
>>> Not sure if this is what you are after, but SPM
>>> <http://sematext.com/spm/> will collect and graph all Storm metrics,
>>> let you do alerting and anomaly detection on them, etc. If you want to
>>> graph custom metrics (e.g. something from your bolts), you can send them in
>>> as custom metrics
>>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
>>> again graph them, alert on them, do anomaly detection on them, stick them
>>> on dashboards, etc. If you want to emit events from your bolts, you can send
>>> events to SPM
>>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
>>> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
>>> can be handy for correlation with alerts and performance graphs when
>>> troubleshooting. Here are some Storm metrics graph:
>>> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>>>
>>>
>>> I hope this helps.
>>>
>>> Otis
>>> --
>>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>>> Solr & Elasticsearch Support * http://sematext.com/
>>>
>>>
>>> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
>>> wrote:
>>>
>>>> Hi,
>>>> Using Storm/Trident, how do I register a metrics consumer to log the
>>>> data I get in the StormUI ?
>>>> I want to look at historical data of my topology, for example the
>>>> execute latency of the topology over time, as this would give me good
>>>> insight as to where things might be going wrong when the system breaks.
>>>>
>>>> I have been following the steps outlined in the BigData CookBook here:
>>>> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>>>>
>>>> However I am not wanting to create my own metrics, instead I just want
>>>> to log the metrics that already exist built in to Storm. It is unclear to
>>>> me how I am supposed to go about doing that.
>>>>
>>>> Thanks
>>>>
>>>> --
>>>> Raphael Hsieh
>>>>
>>>>
>>>>
>>>>
>>>
>>>
>>
>
--
Raphael Hsieh
Re: metrics consumer logging stormUI data
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Gezim,
On Fri, Sep 19, 2014 at 7:27 PM, Gezim Musliaj <gm...@gmail.com> wrote:
> Hey Otis, I was just registered at sematext and I can say that this is
> what I have been looking for.I have just one question, what about the
> delays between the SPM and the Storm Cluster (if they do exist), whats the
> worst case? I mean because these metrics are not calculated locally, but
> using an internet connection.
>
The worst case is that somebody unplugs your servers from the network, but
if that happens you have bigger problems to deal with. In all seriousness,
Storm (local) => SPM (remote/cloud/saas) is not really a problem -- lots of
people successfully use SPM for monitoring Storm, Hadoop, Kafka, and other
types of systems.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
> Thanks !
>
> On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Raphael,
>>
>> Not sure if this is what you are after, but SPM
>> <http://sematext.com/spm/> will collect and graph all Storm metrics, let
>> you do alerting and anomaly detection on them, etc. If you want to graph
>> custom metrics (e.g. something from your bolts), you can send them in as custom
>> metrics
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
>> again graph them, alert on them, do anomaly detection on them, stick them
>> on dashboards, etc. If you want to emit events from your bolts, you can send
>> events to SPM
>> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
>> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
>> can be handy for correlation with alerts and performance graphs when
>> troubleshooting. Here are some Storm metrics graph:
>> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>>
>>
>> I hope this helps.
>>
>> Otis
>> --
>> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
>> Solr & Elasticsearch Support * http://sematext.com/
>>
>>
>> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
>> wrote:
>>
>>> Hi,
>>> Using Storm/Trident, how do I register a metrics consumer to log the
>>> data I get in the StormUI ?
>>> I want to look at historical data of my topology, for example the
>>> execute latency of the topology over time, as this would give me good
>>> insight as to where things might be going wrong when the system breaks.
>>>
>>> I have been following the steps outlined in the BigData CookBook here:
>>> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>>>
>>> However I am not wanting to create my own metrics, instead I just want
>>> to log the metrics that already exist built in to Storm. It is unclear to
>>> me how I am supposed to go about doing that.
>>>
>>> Thanks
>>>
>>> --
>>> Raphael Hsieh
>>>
>>>
>>>
>>>
>>
>>
>
Re: metrics consumer logging stormUI data
Posted by Gezim Musliaj <gm...@gmail.com>.
Hey Otis, I was just registered at sematext and I can say that this is what
I have been looking for.I have just one question, what about the delays
between the SPM and the Storm Cluster (if they do exist), whats the worst
case? I mean because these metrics are not calculated locally, but using an
internet connection.
Thanks !
On Sat, Sep 20, 2014 at 1:15 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Raphael,
>
> Not sure if this is what you are after, but SPM <http://sematext.com/spm/>
> will collect and graph all Storm metrics, let you do alerting and anomaly
> detection on them, etc. If you want to graph custom metrics (e.g.
> something from your bolts), you can send them in as custom metrics
> <https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
> again graph them, alert on them, do anomaly detection on them, stick them
> on dashboards, etc. If you want to emit events from your bolts, you can send
> events to SPM
> <https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
> too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
> can be handy for correlation with alerts and performance graphs when
> troubleshooting. Here are some Storm metrics graph:
> http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
>
>
> I hope this helps.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com>
> wrote:
>
>> Hi,
>> Using Storm/Trident, how do I register a metrics consumer to log the data
>> I get in the StormUI ?
>> I want to look at historical data of my topology, for example the execute
>> latency of the topology over time, as this would give me good insight as to
>> where things might be going wrong when the system breaks.
>>
>> I have been following the steps outlined in the BigData CookBook here:
>> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>>
>> However I am not wanting to create my own metrics, instead I just want to
>> log the metrics that already exist built in to Storm. It is unclear to me
>> how I am supposed to go about doing that.
>>
>> Thanks
>>
>> --
>> Raphael Hsieh
>>
>>
>>
>>
>
>
Re: metrics consumer logging stormUI data
Posted by Otis Gospodnetic <ot...@gmail.com>.
Raphael,
Not sure if this is what you are after, but SPM <http://sematext.com/spm/>
will collect and graph all Storm metrics, let you do alerting and anomaly
detection on them, etc. If you want to graph custom metrics (e.g.
something from your bolts), you can send them in as custom metrics
<https://sematext.atlassian.net/wiki/display/PUBSPM/Custom+Metrics> and
again graph them, alert on them, do anomaly detection on them, stick them
on dashboards, etc. If you want to emit events from your bolts, you can send
events to SPM
<https://sematext.atlassian.net/wiki/display/PUBSPM/Events+Integration>,
too, or you can send them to Logsene <http://www.sematext.com/logsene/>...
can be handy for correlation with alerts and performance graphs when
troubleshooting. Here are some Storm metrics graph:
http://blog.sematext.com/2014/01/30/announcement-apache-storm-monitoring-in-spm/
I hope this helps.
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Fri, Sep 19, 2014 at 6:12 PM, Raphael Hsieh <ra...@gmail.com> wrote:
> Hi,
> Using Storm/Trident, how do I register a metrics consumer to log the data
> I get in the StormUI ?
> I want to look at historical data of my topology, for example the execute
> latency of the topology over time, as this would give me good insight as to
> where things might be going wrong when the system breaks.
>
> I have been following the steps outlined in the BigData CookBook here:
> http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to
>
> However I am not wanting to create my own metrics, instead I just want to
> log the metrics that already exist built in to Storm. It is unclear to me
> how I am supposed to go about doing that.
>
> Thanks
>
> --
> Raphael Hsieh
>
>
>
>
Re: metrics consumer logging stormUI data
Posted by Harsha <st...@harsha.io>.
you can add the following in storm.yaml to enable
LoggingMetricsConsumer
topology.metrics.consumer.register:
- class: "backtype.storm.metric.LoggingMetricsConsumer"
parallelism.hint: 1
storm UI doesn't display user provided metrics and it doesn't
also keep the historical data about the metrics , if the
cluster is restarted topology stats will be reset.
you can find bit more info on this page
[1]http://blog.relateiq.com/monitoring-storm/
-Harsha
On Fri, Sep 19, 2014, at 03:12 PM, Raphael Hsieh wrote:
Hi,
Using Storm/Trident, how do I register a metrics consumer to
log the data I get in the StormUI ?
I want to look at historical data of my topology, for example
the execute latency of the topology over time, as this would
give me good insight as to where things might be going wrong
when the system breaks.
I have been following the steps outlined in the BigData
CookBook
here: [2]http://www.bigdata-cookbook.com/post/72320512609/storm
-metrics-how-to
However I am not wanting to create my own metrics, instead I
just want to log the metrics that already exist built in to
Storm. It is unclear to me how I am supposed to go about doing
that.
Thanks
--
Raphael Hsieh
References
1. http://blog.relateiq.com/monitoring-storm/
2. http://www.bigdata-cookbook.com/post/72320512609/storm-metrics-how-to