You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by bijoy deb <bi...@gmail.com> on 2014/01/20 18:05:27 UTC
How to capture custom Storm UI related metrics using Ganglia
Hi,
I am trying to use Ganglia to capture all the metrics that are displayed
in Storm UI. I am not quite sure on how to do that.I was able to integrate
Cassandra with Ganglia,where I just need to specify the metric name I want
to capture in a cassandra config file and Ganglia displays that metric.Is
there anything similar for Storm as well? I know its possible to capture
the JVM,Memory stats of the Storm cluster,but I am more interested in Storm
specific parameters such as Nimbus config,Number of spouts/bolts,Tuples
emitted,Tuples acked,Latency,etc.
Any help in this regard will be highly appreciated.
Re: How to capture custom Storm UI related metrics using Ganglia
Posted by Mark Greene <ma...@evertrue.com>.
I used a library from Codahale that took care of the pushing to graphite.
Ultimately all it consists of a is a background thread that scans a
ConcurrentHashMap and sends it over the desired protocol (TCP/UDP/HTTP).
I did have to do some segmentation per JVM as I run topologies on multiple
workers. Something to keep in mind.
On Wed, Jan 22, 2014 at 5:23 AM, bijoy deb <bi...@gmail.com>wrote:
> Hi,
> The approach to send metrics from Storm to something else will vary from
> one charting service to another.
> Did you use Ganglia as the charting service? To send the metrics to
> Ganglia from Storm,I am looking for some api that can be used to push the
> metrics to Ganglia server.
>
> Thanks
>
>
> On Tue, Jan 21, 2014 at 7:52 PM, Mark Greene <ma...@evertrue.com> wrote:
>
>> Ah sorry. So I use a singleton that runs a ScheduledThreadExecutor that
>> ships the counters to my charting service.
>>
>>
>> On Tue, Jan 21, 2014 at 12:05 AM, bijoy deb <bi...@gmail.com>wrote:
>>
>>> Hi Mark,
>>>
>>> I have the required code already in place that gathers the Storm metrics
>>> via Nimbus api calls.
>>> What I am looking for is a way to forward those to Ganglia to display on
>>> Ganglia web screen.
>>>
>>> Thanks
>>> Bijoy
>>>
>>>
>>> On Tue, Jan 21, 2014 at 6:31 AM, Mark Greene <ma...@evertrue.com> wrote:
>>>
>>>> Hey Bijoy,
>>>>
>>>> Not sure how you would go about capturing number of spouts/bolts but
>>>> we've done some custom metrics capturing with respect to latencies inside
>>>> spouts and bolts.
>>>>
>>>> For spouts, when you override the open method, you have access to a
>>>> TopologyContext object. This is where you can add a TaskHook. We use this
>>>> as a callback to record latencies for spout.
>>>>
>>>> For bolts similar situation but for the prepare method.
>>>>
>>>> You just also need to be aware that the sampling rate of these callback
>>>> hooks are controlled from within the Topology Config object via conf.
>>>> setStatsSampleRate(double).
>>>>
>>>>
>>>> On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb <bi...@gmail.com>wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I am trying to use Ganglia to capture all the metrics that are
>>>>> displayed in Storm UI. I am not quite sure on how to do that.I was able to
>>>>> integrate Cassandra with Ganglia,where I just need to specify the metric
>>>>> name I want to capture in a cassandra config file and Ganglia displays that
>>>>> metric.Is there anything similar for Storm as well? I know its possible to
>>>>> capture the JVM,Memory stats of the Storm cluster,but I am more interested
>>>>> in Storm specific parameters such as Nimbus config,Number of
>>>>> spouts/bolts,Tuples emitted,Tuples acked,Latency,etc.
>>>>>
>>>>> Any help in this regard will be highly appreciated.
>>>>>
>>>>
>>>>
>>>
>>
>
Re: How to capture custom Storm UI related metrics using Ganglia
Posted by bijoy deb <bi...@gmail.com>.
Hi,
The approach to send metrics from Storm to something else will vary from
one charting service to another.
Did you use Ganglia as the charting service? To send the metrics to Ganglia
from Storm,I am looking for some api that can be used to push the metrics
to Ganglia server.
Thanks
On Tue, Jan 21, 2014 at 7:52 PM, Mark Greene <ma...@evertrue.com> wrote:
> Ah sorry. So I use a singleton that runs a ScheduledThreadExecutor that
> ships the counters to my charting service.
>
>
> On Tue, Jan 21, 2014 at 12:05 AM, bijoy deb <bi...@gmail.com>wrote:
>
>> Hi Mark,
>>
>> I have the required code already in place that gathers the Storm metrics
>> via Nimbus api calls.
>> What I am looking for is a way to forward those to Ganglia to display on
>> Ganglia web screen.
>>
>> Thanks
>> Bijoy
>>
>>
>> On Tue, Jan 21, 2014 at 6:31 AM, Mark Greene <ma...@evertrue.com> wrote:
>>
>>> Hey Bijoy,
>>>
>>> Not sure how you would go about capturing number of spouts/bolts but
>>> we've done some custom metrics capturing with respect to latencies inside
>>> spouts and bolts.
>>>
>>> For spouts, when you override the open method, you have access to a
>>> TopologyContext object. This is where you can add a TaskHook. We use this
>>> as a callback to record latencies for spout.
>>>
>>> For bolts similar situation but for the prepare method.
>>>
>>> You just also need to be aware that the sampling rate of these callback
>>> hooks are controlled from within the Topology Config object via conf.
>>> setStatsSampleRate(double).
>>>
>>>
>>> On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb <bi...@gmail.com>wrote:
>>>
>>>> Hi,
>>>>
>>>> I am trying to use Ganglia to capture all the metrics that are
>>>> displayed in Storm UI. I am not quite sure on how to do that.I was able to
>>>> integrate Cassandra with Ganglia,where I just need to specify the metric
>>>> name I want to capture in a cassandra config file and Ganglia displays that
>>>> metric.Is there anything similar for Storm as well? I know its possible to
>>>> capture the JVM,Memory stats of the Storm cluster,but I am more interested
>>>> in Storm specific parameters such as Nimbus config,Number of
>>>> spouts/bolts,Tuples emitted,Tuples acked,Latency,etc.
>>>>
>>>> Any help in this regard will be highly appreciated.
>>>>
>>>
>>>
>>
>
Re: How to capture custom Storm UI related metrics using Ganglia
Posted by Mark Greene <ma...@evertrue.com>.
Ah sorry. So I use a singleton that runs a ScheduledThreadExecutor that
ships the counters to my charting service.
On Tue, Jan 21, 2014 at 12:05 AM, bijoy deb <bi...@gmail.com>wrote:
> Hi Mark,
>
> I have the required code already in place that gathers the Storm metrics
> via Nimbus api calls.
> What I am looking for is a way to forward those to Ganglia to display on
> Ganglia web screen.
>
> Thanks
> Bijoy
>
>
> On Tue, Jan 21, 2014 at 6:31 AM, Mark Greene <ma...@evertrue.com> wrote:
>
>> Hey Bijoy,
>>
>> Not sure how you would go about capturing number of spouts/bolts but
>> we've done some custom metrics capturing with respect to latencies inside
>> spouts and bolts.
>>
>> For spouts, when you override the open method, you have access to a
>> TopologyContext object. This is where you can add a TaskHook. We use this
>> as a callback to record latencies for spout.
>>
>> For bolts similar situation but for the prepare method.
>>
>> You just also need to be aware that the sampling rate of these callback
>> hooks are controlled from within the Topology Config object via conf.
>> setStatsSampleRate(double).
>>
>>
>> On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb <bi...@gmail.com>wrote:
>>
>>> Hi,
>>>
>>> I am trying to use Ganglia to capture all the metrics that are
>>> displayed in Storm UI. I am not quite sure on how to do that.I was able to
>>> integrate Cassandra with Ganglia,where I just need to specify the metric
>>> name I want to capture in a cassandra config file and Ganglia displays that
>>> metric.Is there anything similar for Storm as well? I know its possible to
>>> capture the JVM,Memory stats of the Storm cluster,but I am more interested
>>> in Storm specific parameters such as Nimbus config,Number of
>>> spouts/bolts,Tuples emitted,Tuples acked,Latency,etc.
>>>
>>> Any help in this regard will be highly appreciated.
>>>
>>
>>
>
Re: How to capture custom Storm UI related metrics using Ganglia
Posted by bijoy deb <bi...@gmail.com>.
Hi Mark,
I have the required code already in place that gathers the Storm metrics
via Nimbus api calls.
What I am looking for is a way to forward those to Ganglia to display on
Ganglia web screen.
Thanks
Bijoy
On Tue, Jan 21, 2014 at 6:31 AM, Mark Greene <ma...@evertrue.com> wrote:
> Hey Bijoy,
>
> Not sure how you would go about capturing number of spouts/bolts but we've
> done some custom metrics capturing with respect to latencies inside spouts
> and bolts.
>
> For spouts, when you override the open method, you have access to a
> TopologyContext object. This is where you can add a TaskHook. We use this
> as a callback to record latencies for spout.
>
> For bolts similar situation but for the prepare method.
>
> You just also need to be aware that the sampling rate of these callback
> hooks are controlled from within the Topology Config object via conf.
> setStatsSampleRate(double).
>
>
> On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb <bi...@gmail.com>wrote:
>
>> Hi,
>>
>> I am trying to use Ganglia to capture all the metrics that are displayed
>> in Storm UI. I am not quite sure on how to do that.I was able to integrate
>> Cassandra with Ganglia,where I just need to specify the metric name I want
>> to capture in a cassandra config file and Ganglia displays that metric.Is
>> there anything similar for Storm as well? I know its possible to capture
>> the JVM,Memory stats of the Storm cluster,but I am more interested in Storm
>> specific parameters such as Nimbus config,Number of spouts/bolts,Tuples
>> emitted,Tuples acked,Latency,etc.
>>
>> Any help in this regard will be highly appreciated.
>>
>
>
Re: How to capture custom Storm UI related metrics using Ganglia
Posted by Mark Greene <ma...@evertrue.com>.
Hey Bijoy,
Not sure how you would go about capturing number of spouts/bolts but we've
done some custom metrics capturing with respect to latencies inside spouts
and bolts.
For spouts, when you override the open method, you have access to a
TopologyContext object. This is where you can add a TaskHook. We use this
as a callback to record latencies for spout.
For bolts similar situation but for the prepare method.
You just also need to be aware that the sampling rate of these callback
hooks are controlled from within the Topology Config object via conf.
setStatsSampleRate(double).
On Mon, Jan 20, 2014 at 12:05 PM, bijoy deb <bi...@gmail.com>wrote:
> Hi,
>
> I am trying to use Ganglia to capture all the metrics that are displayed
> in Storm UI. I am not quite sure on how to do that.I was able to integrate
> Cassandra with Ganglia,where I just need to specify the metric name I want
> to capture in a cassandra config file and Ganglia displays that metric.Is
> there anything similar for Storm as well? I know its possible to capture
> the JVM,Memory stats of the Storm cluster,but I am more interested in Storm
> specific parameters such as Nimbus config,Number of spouts/bolts,Tuples
> emitted,Tuples acked,Latency,etc.
>
> Any help in this regard will be highly appreciated.
>