Re: Storm monitoring

[Otis Gospodnetić <ot...@gmail.com>]

For anyone interested in Storm monitoring and not interested in the DIY
approach, SPM has nice built-in Storm metrics monitoring/charting/alerting:
https://sematext.com/spm/

Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/


On Tue, May 31, 2016 at 9:19 AM, Matthew Lowe <gi...@gmail.com>
wrote:

> Thought you all might be interested.
>
> I have now got capacity monitoring working with AWS Cloudwatch. These are
> 2 bolts within the same topology:
>
> ​
>
>
> #! /bin/bash
>
> API_KEY='XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX'
>
> # get all the topology id's
> TOPOLOGY_IDS=`curl -s
> http://XXXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/summary | json_pp |
> grep '"id"' | sed 's/.* : "\(.*\)".*/\1/'`
> for id in ${TOPOLOGY_IDS} ; do
> TOPOLOGY=`curl -s
> http://XXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/${id}?sys=false`
> <http://XXXXXXXXXXXXXXXXXXXXXXX:8080/api/v1/topology/$%7Bid%7D?sys=false>
>
> # check the capacity for each task
> CAPACITIES=`echo ${TOPOLOGY} | json_pp | grep '"capacity"' | sed 's/.* :
> "\(.*\)".*/\1/'`
> CAPACITIES_ARRAY=()
> for capactiy in ${CAPACITIES} ; do
> CAPACITIES_ARRAY+=("${capactiy}")
> done
>
> # get all of the bolt ids
> BOLT_IDS=`echo ${TOPOLOGY} | json_pp | grep '"boltId"' | sed 's/.* :
> "\(.*\)".*/\1/'`
> BOLT_IDS_ARRAY=()
> for boltId in ${BOLT_IDS} ; do
> BOLT_IDS_ARRAY+=("${boltId}")
> done
>
>     # send a trigger if needed
>     # capacity and bolt id arrays will always be the same size
> for(( x=0; x<${#CAPACITIES_ARRAY[@]}; x++ ));  do
> TIMESTAMP=`date +"%Y-%m-%dT%H:%M:%S.%N"`
> /usr/local/bin/aws cloudwatch put-metric-data --metric-name
> "${BOLT_IDS_ARRAY[$x]}-capacity" --namespace "${id}" --value
> "${CAPACITIES_ARRAY[$x]}" --timestamp "${TIMESTAMP}"
> done
>
>
> done
>
> On Mon, May 30, 2016 at 5:37 PM, Radhwane Chebaane <
> r.chebaane@mindlytix.com> wrote:
>
>>
>> Using graphite API was so helpful for us, but since it doesn't support
>> *Tags* introduced since *InfluxDB 0.9*, we created a *new metric
>> Consumer <https://github.com/mathieuboniface/storm-metrics-influxdb>*
>> that supports the new InfluxDB API instead of Grahpite API. *Tags* are
>> important when filtering dashboard metrics based on components, bolt or
>> worker name.
>>
>> Regards,
>> Radhwane
>>
>>
>> On 30/05/2016 17:19, Stephen Powis wrote:
>>
>> +1 for graphite and grafana via Verisign's plugin.
>>
>> Using graphite a few years ago was a real game changer for us, and more
>> recently grafana to help build out dashboards instead of copy/pasting
>> graphite urls around.  Here's two different dashboards we have relating to
>> our storm topologies.  We're able to correlate information from all parts
>> of our app, hardware monitoring metrics (via zabbix) and of course storm.
>> Additionally we use seyren on top of graphite for our alerting as well.
>>
>> bolt specific dashboard <http://i.imgur.com/ftKtci5.png>
>>
>> Dashboard correlating lots of related information from various sources
>> <http://i.imgur.com/t7yJ8d5.jpg>
>>
>>
>>
>> On Mon, May 30, 2016 at 9:13 AM, Radhwane Chebaane <
>> <r....@mindlytix.com> wrote:
>>
>>> Hi Matthew,
>>>
>>> We actually use the InfluxData <https://influxdata.com/> Stack
>>> (InfluxDb + Grafana).
>>> We send our data directly to a time-series database, *InfluxDB*
>>> <https://influxdata.com/time-series-platform/influxdb/>. Then, we
>>> visualize metrics with a customizable dashboard, *Grafana*
>>> <http://grafana.org/>.
>>> This way, you can have real-time metrics on your Storm topology. You may
>>> also add custom metrics for enhanced monitoring.
>>>
>>> To export Storm metrics to InfluxDB you can use this *MetricsConsumer *which
>>> is compatible with the latest version of InfluxDB and Storm 1.0.0:
>>> https://github.com/mathieuboniface/storm-metrics-influxdb
>>>
>>> Or you can use the old Verisign plug-in with Graphite protocol:
>>> https://github.com/verisign/storm-graphite
>>>
>>> Best regards,
>>> Radhwane CHEBAANE
>>>
>>>
>>> On 30/05/2016 14:47, Matthew Lowe wrote:
>>>
>>> Hello all.
>>>
>>> What kind of monitoring solutions do you use with storm?
>>>
>>> For example I have a bash script that reads the Json data from the REST UI and alerts if there are any bolts with high capacities.
>>>
>>> It's only small and hacky, but I am genuinely interested to how you all monitor your topologies.
>>>
>>> Best Regards
>>> Matthew Lowe
>>>
>>>
>>>
>>
>>
>