You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@storm.apache.org by Kevin Conaway <ke...@gmail.com> on 2016/05/01 14:17:01 UTC
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
One thing I would like to see added (if not already present) is the ability
to register metrics that are not tied to a component.
As of now, the only non-component metrics are reported by the SystemBolt
pseudo-component which feels like a work-around. It reports JVM level
metrics like GC time, heap size and other things that aren't associated
with a given component.
It would be great if application developers could expose similar metrics
like this for things like connection pools and other JVM wide objects that
aren't unique to a specific component.
I don't think this is possible now, is it?
On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com> wrote:
> Let me start sharing my thought. :)
>
> 1. Need to enrich docs about metrics / stats.
>
> In fact, I couldn't see the fact - topology stats are sampled by default
> and sample rate is 0.05 - from the docs when I was newbie of Apache
> Storm. It made me misleading and made me saying "Why there're difference
> between the counts?". I also saw some mails from user@ about same question.
> If we include this to guide doc that would be better.
>
> And Metrics document page
> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
> written. I think it has appropriate headings but lacks contents on each
> heading.
> It should be addressed, and introducing some external metrics consumer
> plugins (like storm-graphite <https://github.com/verisign/storm-graphite> from
> Verisign) would be great, too.
>
> 2. Need to increase sample rate or (ideally) no sampling at all.
>
> Let's postpone considering performance hit at this time.
> Ideally, we expect precision of metrics gets better when we increase
> sample rate. It affects non-gauge kinds of metrics which are counter, and
> latency, and so on.
>
> Btw, I would like to hear about opinions on latency since I'm not an
> expert.
> Storm provides only average latency and it's indeed based on sample rate.
> Do we feel OK with this? If not how much having also percentiles can help
> us?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
>> Hi Storm users,
>>
>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>
>> If you subscribed dev@ mailing list, you may have seen that recently
>> we're addressing the metrics feature on Apache Storm.
>>
>> For now, improvements are going forward based on current metrics feature.
>>
>> - Improve (Topology) MetricsConsumer
>> <https://issues.apache.org/jira/browse/STORM-1699>
>> - Provide topology metrics in detail (metrics per each stream)
>> <https://issues.apache.org/jira/browse/STORM-1719>
>> - (WIP) Introduce Cluster Metrics Consumer
>>
>> As I don't maintain large cluster for myself, I really want to collect
>> the any ideas for improving, any inconveniences, use cases of Metrics with
>> community members, so we're on the right way to go forward.
>>
>> Let's talk!
>>
>> Thanks in advance,
>> Jungtaek Lim (HeartSaVioR)
>>
>
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
Posted by Jungtaek Lim <ka...@gmail.com>.
Harsha,
Yes sure, I applied regex for filtering by metric name. Please refer
relevant unit test
<https://github.com/HeartSaVioR/storm/blob/STORM-1700/storm-core/test/jvm/org/apache/storm/metric/filter/FilterByMetricNameTest.java>
from pull request <https://github.com/apache/storm/pull/1324>.
I'm curious other features (component name, host, etc) could also be useful
for filtering. If we think they're useful, it could be easy to add, too. Also
option could be expanded to take a list of filter configurations, finally
making the filter list.
Just would like to see the opinions whether it's better to keep it simple,
or give full flexibilities.
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 5월 3일 (화) 오후 2:47, Harsha <st...@harsha.io>님이 작성:
> Jungtaek,
> I think filters that can support a regex gives more
> felxibility.
> Thanks,
> Harsha
>
>
> On Mon, May 2, 2016, at 07:48 PM, Jungtaek Lim wrote:
>
> Kevin,
>
> For specific task, you can register your own metrics which resides per
> task.
> But metrics doc on Storm is not kind enough to let users follow, so I
> addressed this and submitted pull request.
> https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md
>
> There're no custom worker metrics for users since Storm abstracts user's
> logic into 'task' so normally users don't need to measure JVM level metrics
> (except JMX metrics). But it would be possible to add if it's really great.
>
> Does current Storm work for you now? Or we would like to address / improve
> something?
>
> Stephen and Harsha,
>
> Improve MetricsConsumer is an umbrella issue, and it contains the feature 'blacklist
> & whitelist of metrics <https://issues.apache.org/jira/browse/STORM-1700>
> '.
> For now it filters metrics by only metrics name, but adding filter targets
> to component name, host, etc. are easy so I would like to see the needs
> before making the change.
> Do you think adding filter targets makes sense? Or we can just start with
> metrics name filter?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 5월 3일 (화) 오전 2:18, Harsha <st...@harsha.io>님이 작성:
>
>
> Jungtaek,
> Probably a filter config to whitelist and blacklist certain
> metrics. So that it will scale if there are too many workers and users can
> turn off certain metrics.
>
> Thanks,
> Harsha
>
>
> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>
> Oooh I'd love this as well! I really dig the ease of the metric framework
> in storm and have all the metrics go thru one centralized config. But as
> the number of storm hosts and number of tasks grow, I've found that
> Graphite/Grafana has a hard time collecting up all the relevant metrics
> across a lot of wildcarded keys for things like hostnames and taskIds to
> properly display my graphs.
>
> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway <ke...@gmail.com>
> wrote:
>
> One thing I would like to see added (if not already present) is the
> ability to register metrics that are not tied to a component.
>
> As of now, the only non-component metrics are reported by the SystemBolt
> pseudo-component which feels like a work-around. It reports JVM level
> metrics like GC time, heap size and other things that aren't associated
> with a given component.
>
> It would be great if application developers could expose similar metrics
> like this for things like connection pools and other JVM wide objects that
> aren't unique to a specific component.
>
> I don't think this is possible now, is it?
>
> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Let me start sharing my thought. :)
>
> 1. Need to enrich docs about metrics / stats.
>
> In fact, I couldn't see the fact - topology stats are sampled by default
> and sample rate is 0.05 - from the docs when I was newbie of Apache
> Storm. It made me misleading and made me saying "Why there're difference
> between the counts?". I also saw some mails from user@ about same question.
> If we include this to guide doc that would be better.
>
> And Metrics document page
> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
> written. I think it has appropriate headings but lacks contents on each
> heading.
> It should be addressed, and introducing some external metrics consumer
> plugins (like storm-graphite <https://github.com/verisign/storm-graphite> from
> Verisign) would be great, too.
>
> 2. Need to increase sample rate or (ideally) no sampling at all.
>
> Let's postpone considering performance hit at this time.
> Ideally, we expect precision of metrics gets better when we increase
> sample rate. It affects non-gauge kinds of metrics which are counter, and
> latency, and so on.
>
> Btw, I would like to hear about opinions on latency since I'm not an
> expert.
> Storm provides only average latency and it's indeed based on sample rate.
> Do we feel OK with this? If not how much having also percentiles can help
> us?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> Hi Storm users,
>
> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>
> If you subscribed dev@ mailing list, you may have seen that recently
> we're addressing the metrics feature on Apache Storm.
>
> For now, improvements are going forward based on current metrics feature.
>
> - Improve (Topology) MetricsConsumer
> <https://issues.apache.org/jira/browse/STORM-1699>
> - Provide topology metrics in detail (metrics per each stream)
> <https://issues.apache.org/jira/browse/STORM-1719>
> - (WIP) Introduce Cluster Metrics Consumer
>
> As I don't maintain large cluster for myself, I really want to collect the
> any ideas for improving, any inconveniences, use cases of Metrics with
> community members, so we're on the right way to go forward.
>
> Let's talk!
>
> Thanks in advance,
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>
>
>
>
>
>
Re: [DISCUSS] Would like to make collective intelligence about Metrics
on Storm
Posted by Harsha <st...@harsha.io>.
Jungtaek,
I think filters that can support a regex gives more felxibility.
Thanks,
Harsha
On Mon, May 2, 2016, at 07:48 PM, Jungtaek Lim wrote:
> Kevin,
>
> For specific task, you can register your own metrics which resides
> per task.
> But metrics doc on Storm is not kind enough to let users follow, so I
> addressed this and submitted pull request.
> https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md
>
> There're no custom worker metrics for users since Storm abstracts
> user's logic into 'task' so normally users don't need to measure JVM
> level metrics (except JMX metrics). But it would be possible to add if
> it's really great.
>
> Does current Storm work for you now? Or we would like to address /
> improve something?
>
> Stephen and Harsha,
>
> Improve MetricsConsumer is an umbrella issue, and it contains the
> feature 'blacklist & whitelist of metrics[1]'.
> For now it filters metrics by only metrics name, but adding filter
> targets to component name, host, etc. are easy so I would like to see
> the needs before making the change.
> Do you think adding filter targets makes sense? Or we can just start
> with metrics name filter?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 5월 3일 (화) 오전 2:18, Harsha <st...@harsha.io>님이 작성:
>> __
>> Jungtaek,
>> Probably a filter config to whitelist and blacklist certain metrics.
>> So that it will scale if there are too many workers and users can
>> turn off certain metrics.
>>
>> Thanks,
>> Harsha
>>
>>
>> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>>> Oooh I'd love this as well! I really dig the ease of the metric
>>> framework in storm and have all the metrics go thru one centralized
>>> config. But as the number of storm hosts and number of tasks grow,
>>> I've found that Graphite/Grafana has a hard time collecting up all
>>> the relevant metrics across a lot of wildcarded keys for things like
>>> hostnames and taskIds to properly display my graphs.
>>>
>>> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway
>>> <ke...@gmail.com> wrote:
>>>> One thing I would like to see added (if not already present) is the
>>>> ability to register metrics that are not tied to a component.
>>>>
>>>> As of now, the only non-component metrics are reported by the
>>>> SystemBolt pseudo-component which feels like a work-around. It
>>>> reports JVM level metrics like GC time, heap size and other things
>>>> that aren't associated with a given component.
>>>>
>>>> It would be great if application developers could expose similar
>>>> metrics like this for things like connection pools and other JVM
>>>> wide objects that aren't unique to a specific component.
>>>>
>>>> I don't think this is possible now, is it?
>>>>
>>>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com>
>>>> wrote:
>>>>> Let me start sharing my thought. :)
>>>>>
>>>>> 1. Need to enrich docs about metrics / stats.
>>>>>
>>>>> In fact, I couldn't see the fact - topology stats are sampled by
>>>>> default and sample rate is 0.05 - from the docs when I was newbie
>>>>> of Apache Storm. It made me misleading and made me saying "Why
>>>>> there're difference between the counts?". I also saw some mails
>>>>> from user@ about same question. If we include this to guide doc
>>>>> that would be better.
>>>>>
>>>>> And Metrics document page[2] seems not well written. I think it
>>>>> has appropriate headings but lacks contents on each heading.
>>>>> It should be addressed, and introducing some external metrics
>>>>> consumer plugins (like storm-graphite[3] from Verisign) would be
>>>>> great, too.
>>>>>
>>>>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>>>>
>>>>> Let's postpone considering performance hit at this time.
>>>>> Ideally, we expect precision of metrics gets better when we
>>>>> increase sample rate. It affects non-gauge kinds of metrics which
>>>>> are counter, and latency, and so on.
>>>>>
>>>>> Btw, I would like to hear about opinions on latency since I'm not
>>>>> an expert.
>>>>> Storm provides only average latency and it's indeed based on
>>>>> sample rate. Do we feel OK with this? If not how much having also
>>>>> percentiles can help us?
>>>>>
>>>>> Thanks,
>>>>> Jungtaek Lim (HeartSaVioR)
>>>>>
>>>>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>>>> Hi Storm users,
>>>>>>
>>>>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>>>>>
>>>>>> If you subscribed dev@ mailing list, you may have seen that
>>>>>> recently we're addressing the metrics feature on Apache Storm.
>>>>>>
>>>>>> For now, improvements are going forward based on current metrics
>>>>>> feature.
>>>>>>
>>>>>> - Improve (Topology) MetricsConsumer[4]
>>>>>> - Provide topology metrics in detail (metrics per each stream)[5]
>>>>>> - (WIP) Introduce Cluster Metrics Consumer
>>>>>>
>>>>>> As I don't maintain large cluster for myself, I really want to
>>>>>> collect the any ideas for improving, any inconveniences, use
>>>>>> cases of Metrics with community members, so we're on the right
>>>>>> way to go forward.
>>>>>>
>>>>>> Let's talk!
>>>>>>
>>>>>> Thanks in advance,
>>>>>> Jungtaek Lim (HeartSaVioR)
>>>>
>>>>
>>>>
>>>> --
>>>> Kevin Conaway http://www.linkedin.com/pub/kevin-conaway/7/107/580/
>>>> https://github.com/kevinconaway
>>>>
>>
Links:
1. https://issues.apache.org/jira/browse/STORM-1700
2. http://storm.apache.org/releases/1.0.0/Metrics.html
3. https://github.com/verisign/storm-graphite
4. https://issues.apache.org/jira/browse/STORM-1699
5. https://issues.apache.org/jira/browse/STORM-1719
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
Posted by Jungtaek Lim <ka...@gmail.com>.
Kevin,
It makes sense but I can't imagine beautiful way to address that case.
If we change the view from task to worker, IMO worker level metrics
shouldn't look up task level objects, cause it's beyond the layer, and
worker doesn't even know what instances have what fields.
If there's a way to access worker level instance without any references on
tasks, there's other way to collect that metrics, using SystemBolt.
SystemBolt also registers Storm options 'worker.metrics' (for applying
whole workers in cluster) and 'topology.worker.metrics' (for applying whole
workers in this topology) which are map of pair (metrics name, metric
class). Since they will be instantiated and registered dynamically in
SystemBolt.prepare() and users can't access instances directly, it should
be kind of gauge.
(Metrics doc doesn't describe this feature and I also missed it. I'll add
this to STORM-1724 <https://issues.apache.org/jira/browse/STORM-1724>, too.)
But that case you want to address is different from this, worker level
instance shared across multiple tasks and it should be accessed from the
task. In this case, I can't think nicer way for now and we still rely on
metrics per task.
Do you have some ideas or thought?
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 5월 4일 (수) 오후 9:23, Kevin Conaway <ke...@gmail.com>님이 작성:
> >For specific task, you can register your own metrics which resides per
> task.
>
> Exactly, thats the problem. For something like a JDBC connection pool (or
> a Cassandra cluster/session), its not tied to any one task or component, it
> is usually shared amongst all tasks in the JVM. Reporting on those objects
> is difficult and clumsy when they are tied to a semi-unrelated, or random,
> component.
>
> >Or we would like to address / improve something?
>
> It would be nice to be able to register metrics outside of the task
> prepare lifecycle, perhaps at the "worker" level.
>
> Does that make sense?
>
>
> On Mon, May 2, 2016 at 10:48 PM, Jungtaek Lim <ka...@gmail.com> wrote:
>
>> Kevin,
>>
>> For specific task, you can register your own metrics which resides per
>> task.
>> But metrics doc on Storm is not kind enough to let users follow, so I
>> addressed this and submitted pull request.
>> https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md
>>
>> There're no custom worker metrics for users since Storm abstracts user's
>> logic into 'task' so normally users don't need to measure JVM level metrics
>> (except JMX metrics). But it would be possible to add if it's really great.
>>
>> Does current Storm work for you now? Or we would like to address /
>> improve something?
>>
>> Stephen and Harsha,
>>
>> Improve MetricsConsumer is an umbrella issue, and it contains the feature
>> 'blacklist & whitelist of metrics
>> <https://issues.apache.org/jira/browse/STORM-1700>'.
>> For now it filters metrics by only metrics name, but adding filter
>> targets to component name, host, etc. are easy so I would like to see the
>> needs before making the change.
>> Do you think adding filter targets makes sense? Or we can just start with
>> metrics name filter?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>> 2016년 5월 3일 (화) 오전 2:18, Harsha <st...@harsha.io>님이 작성:
>>
>>> Jungtaek,
>>> Probably a filter config to whitelist and blacklist
>>> certain metrics. So that it will scale if there are too many workers and
>>> users can turn off certain metrics.
>>>
>>> Thanks,
>>> Harsha
>>>
>>>
>>> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>>>
>>> Oooh I'd love this as well! I really dig the ease of the metric
>>> framework in storm and have all the metrics go thru one centralized
>>> config. But as the number of storm hosts and number of tasks grow, I've
>>> found that Graphite/Grafana has a hard time collecting up all the relevant
>>> metrics across a lot of wildcarded keys for things like hostnames and
>>> taskIds to properly display my graphs.
>>>
>>> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway <kevin.a.conaway@gmail.com
>>> > wrote:
>>>
>>> One thing I would like to see added (if not already present) is the
>>> ability to register metrics that are not tied to a component.
>>>
>>> As of now, the only non-component metrics are reported by the SystemBolt
>>> pseudo-component which feels like a work-around. It reports JVM level
>>> metrics like GC time, heap size and other things that aren't associated
>>> with a given component.
>>>
>>> It would be great if application developers could expose similar metrics
>>> like this for things like connection pools and other JVM wide objects that
>>> aren't unique to a specific component.
>>>
>>> I don't think this is possible now, is it?
>>>
>>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com>
>>> wrote:
>>>
>>> Let me start sharing my thought. :)
>>>
>>> 1. Need to enrich docs about metrics / stats.
>>>
>>> In fact, I couldn't see the fact - topology stats are sampled by default
>>> and sample rate is 0.05 - from the docs when I was newbie of Apache
>>> Storm. It made me misleading and made me saying "Why there're
>>> difference between the counts?". I also saw some mails from user@ about
>>> same question. If we include this to guide doc that would be better.
>>>
>>> And Metrics document page
>>> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
>>> written. I think it has appropriate headings but lacks contents on each
>>> heading.
>>> It should be addressed, and introducing some external metrics consumer
>>> plugins (like storm-graphite
>>> <https://github.com/verisign/storm-graphite> from Verisign) would be
>>> great, too.
>>>
>>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>>
>>> Let's postpone considering performance hit at this time.
>>> Ideally, we expect precision of metrics gets better when we increase
>>> sample rate. It affects non-gauge kinds of metrics which are counter,
>>> and latency, and so on.
>>>
>>> Btw, I would like to hear about opinions on latency since I'm not an
>>> expert.
>>> Storm provides only average latency and it's indeed based on sample
>>> rate. Do we feel OK with this? If not how much having also percentiles can
>>> help us?
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>
>>> Hi Storm users,
>>>
>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>>
>>> If you subscribed dev@ mailing list, you may have seen that recently
>>> we're addressing the metrics feature on Apache Storm.
>>>
>>> For now, improvements are going forward based on current metrics feature.
>>>
>>> - Improve (Topology) MetricsConsumer
>>> <https://issues.apache.org/jira/browse/STORM-1699>
>>> - Provide topology metrics in detail (metrics per each stream)
>>> <https://issues.apache.org/jira/browse/STORM-1719>
>>> - (WIP) Introduce Cluster Metrics Consumer
>>>
>>> As I don't maintain large cluster for myself, I really want to collect
>>> the any ideas for improving, any inconveniences, use cases of Metrics with
>>> community members, so we're on the right way to go forward.
>>>
>>> Let's talk!
>>>
>>> Thanks in advance,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>>
>>>
>>>
>>> --
>>> Kevin Conaway
>>> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
>>> https://github.com/kevinconaway
>>>
>>>
>>>
>>>
>>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
Posted by Kevin Conaway <ke...@gmail.com>.
>For specific task, you can register your own metrics which resides per
task.
Exactly, thats the problem. For something like a JDBC connection pool (or
a Cassandra cluster/session), its not tied to any one task or component, it
is usually shared amongst all tasks in the JVM. Reporting on those objects
is difficult and clumsy when they are tied to a semi-unrelated, or random,
component.
>Or we would like to address / improve something?
It would be nice to be able to register metrics outside of the task prepare
lifecycle, perhaps at the "worker" level.
Does that make sense?
On Mon, May 2, 2016 at 10:48 PM, Jungtaek Lim <ka...@gmail.com> wrote:
> Kevin,
>
> For specific task, you can register your own metrics which resides per
> task.
> But metrics doc on Storm is not kind enough to let users follow, so I
> addressed this and submitted pull request.
> https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md
>
> There're no custom worker metrics for users since Storm abstracts user's
> logic into 'task' so normally users don't need to measure JVM level metrics
> (except JMX metrics). But it would be possible to add if it's really great.
>
> Does current Storm work for you now? Or we would like to address / improve
> something?
>
> Stephen and Harsha,
>
> Improve MetricsConsumer is an umbrella issue, and it contains the feature 'blacklist
> & whitelist of metrics <https://issues.apache.org/jira/browse/STORM-1700>
> '.
> For now it filters metrics by only metrics name, but adding filter targets
> to component name, host, etc. are easy so I would like to see the needs
> before making the change.
> Do you think adding filter targets makes sense? Or we can just start with
> metrics name filter?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
>
> 2016년 5월 3일 (화) 오전 2:18, Harsha <st...@harsha.io>님이 작성:
>
>> Jungtaek,
>> Probably a filter config to whitelist and blacklist
>> certain metrics. So that it will scale if there are too many workers and
>> users can turn off certain metrics.
>>
>> Thanks,
>> Harsha
>>
>>
>> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>>
>> Oooh I'd love this as well! I really dig the ease of the metric
>> framework in storm and have all the metrics go thru one centralized
>> config. But as the number of storm hosts and number of tasks grow, I've
>> found that Graphite/Grafana has a hard time collecting up all the relevant
>> metrics across a lot of wildcarded keys for things like hostnames and
>> taskIds to properly display my graphs.
>>
>> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway <ke...@gmail.com>
>> wrote:
>>
>> One thing I would like to see added (if not already present) is the
>> ability to register metrics that are not tied to a component.
>>
>> As of now, the only non-component metrics are reported by the SystemBolt
>> pseudo-component which feels like a work-around. It reports JVM level
>> metrics like GC time, heap size and other things that aren't associated
>> with a given component.
>>
>> It would be great if application developers could expose similar metrics
>> like this for things like connection pools and other JVM wide objects that
>> aren't unique to a specific component.
>>
>> I don't think this is possible now, is it?
>>
>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>>
>> Let me start sharing my thought. :)
>>
>> 1. Need to enrich docs about metrics / stats.
>>
>> In fact, I couldn't see the fact - topology stats are sampled by default
>> and sample rate is 0.05 - from the docs when I was newbie of Apache
>> Storm. It made me misleading and made me saying "Why there're difference
>> between the counts?". I also saw some mails from user@ about same question.
>> If we include this to guide doc that would be better.
>>
>> And Metrics document page
>> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
>> written. I think it has appropriate headings but lacks contents on each
>> heading.
>> It should be addressed, and introducing some external metrics consumer
>> plugins (like storm-graphite <https://github.com/verisign/storm-graphite>
>> from Verisign) would be great, too.
>>
>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>
>> Let's postpone considering performance hit at this time.
>> Ideally, we expect precision of metrics gets better when we increase
>> sample rate. It affects non-gauge kinds of metrics which are counter,
>> and latency, and so on.
>>
>> Btw, I would like to hear about opinions on latency since I'm not an
>> expert.
>> Storm provides only average latency and it's indeed based on sample rate.
>> Do we feel OK with this? If not how much having also percentiles can help
>> us?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>
>> Hi Storm users,
>>
>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>
>> If you subscribed dev@ mailing list, you may have seen that recently
>> we're addressing the metrics feature on Apache Storm.
>>
>> For now, improvements are going forward based on current metrics feature.
>>
>> - Improve (Topology) MetricsConsumer
>> <https://issues.apache.org/jira/browse/STORM-1699>
>> - Provide topology metrics in detail (metrics per each stream)
>> <https://issues.apache.org/jira/browse/STORM-1719>
>> - (WIP) Introduce Cluster Metrics Consumer
>>
>> As I don't maintain large cluster for myself, I really want to collect
>> the any ideas for improving, any inconveniences, use cases of Metrics with
>> community members, so we're on the right way to go forward.
>>
>> Let's talk!
>>
>> Thanks in advance,
>> Jungtaek Lim (HeartSaVioR)
>>
>>
>>
>>
>> --
>> Kevin Conaway
>> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
>> https://github.com/kevinconaway
>>
>>
>>
>>
>
--
Kevin Conaway
http://www.linkedin.com/pub/kevin-conaway/7/107/580/
https://github.com/kevinconaway
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
Posted by Jungtaek Lim <ka...@gmail.com>.
Kevin,
For specific task, you can register your own metrics which resides per
task.
But metrics doc on Storm is not kind enough to let users follow, so I
addressed this and submitted pull request.
https://github.com/HeartSaVioR/storm/blob/STORM-1724-1.x/docs/Metrics.md
There're no custom worker metrics for users since Storm abstracts user's
logic into 'task' so normally users don't need to measure JVM level metrics
(except JMX metrics). But it would be possible to add if it's really great.
Does current Storm work for you now? Or we would like to address / improve
something?
Stephen and Harsha,
Improve MetricsConsumer is an umbrella issue, and it contains the
feature 'blacklist
& whitelist of metrics <https://issues.apache.org/jira/browse/STORM-1700>'.
For now it filters metrics by only metrics name, but adding filter targets
to component name, host, etc. are easy so I would like to see the needs
before making the change.
Do you think adding filter targets makes sense? Or we can just start with
metrics name filter?
Thanks,
Jungtaek Lim (HeartSaVioR)
2016년 5월 3일 (화) 오전 2:18, Harsha <st...@harsha.io>님이 작성:
> Jungtaek,
> Probably a filter config to whitelist and blacklist certain
> metrics. So that it will scale if there are too many workers and users can
> turn off certain metrics.
>
> Thanks,
> Harsha
>
>
> On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
>
> Oooh I'd love this as well! I really dig the ease of the metric framework
> in storm and have all the metrics go thru one centralized config. But as
> the number of storm hosts and number of tasks grow, I've found that
> Graphite/Grafana has a hard time collecting up all the relevant metrics
> across a lot of wildcarded keys for things like hostnames and taskIds to
> properly display my graphs.
>
> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway <ke...@gmail.com>
> wrote:
>
> One thing I would like to see added (if not already present) is the
> ability to register metrics that are not tied to a component.
>
> As of now, the only non-component metrics are reported by the SystemBolt
> pseudo-component which feels like a work-around. It reports JVM level
> metrics like GC time, heap size and other things that aren't associated
> with a given component.
>
> It would be great if application developers could expose similar metrics
> like this for things like connection pools and other JVM wide objects that
> aren't unique to a specific component.
>
> I don't think this is possible now, is it?
>
> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
> Let me start sharing my thought. :)
>
> 1. Need to enrich docs about metrics / stats.
>
> In fact, I couldn't see the fact - topology stats are sampled by default
> and sample rate is 0.05 - from the docs when I was newbie of Apache
> Storm. It made me misleading and made me saying "Why there're difference
> between the counts?". I also saw some mails from user@ about same question.
> If we include this to guide doc that would be better.
>
> And Metrics document page
> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
> written. I think it has appropriate headings but lacks contents on each
> heading.
> It should be addressed, and introducing some external metrics consumer
> plugins (like storm-graphite <https://github.com/verisign/storm-graphite> from
> Verisign) would be great, too.
>
> 2. Need to increase sample rate or (ideally) no sampling at all.
>
> Let's postpone considering performance hit at this time.
> Ideally, we expect precision of metrics gets better when we increase
> sample rate. It affects non-gauge kinds of metrics which are counter, and
> latency, and so on.
>
> Btw, I would like to hear about opinions on latency since I'm not an
> expert.
> Storm provides only average latency and it's indeed based on sample rate.
> Do we feel OK with this? If not how much having also percentiles can help
> us?
>
> Thanks,
> Jungtaek Lim (HeartSaVioR)
>
> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>
> Hi Storm users,
>
> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>
> If you subscribed dev@ mailing list, you may have seen that recently
> we're addressing the metrics feature on Apache Storm.
>
> For now, improvements are going forward based on current metrics feature.
>
> - Improve (Topology) MetricsConsumer
> <https://issues.apache.org/jira/browse/STORM-1699>
> - Provide topology metrics in detail (metrics per each stream)
> <https://issues.apache.org/jira/browse/STORM-1719>
> - (WIP) Introduce Cluster Metrics Consumer
>
> As I don't maintain large cluster for myself, I really want to collect the
> any ideas for improving, any inconveniences, use cases of Metrics with
> community members, so we're on the right way to go forward.
>
> Let's talk!
>
> Thanks in advance,
> Jungtaek Lim (HeartSaVioR)
>
>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>
>
>
>
Re: [DISCUSS] Would like to make collective intelligence about Metrics
on Storm
Posted by Harsha <st...@harsha.io>.
Jungtaek,
Probably a filter config to whitelist and blacklist certain metrics. So
that it will scale if there are too many workers and users can turn off
certain metrics.
Thanks,
Harsha
On Mon, May 2, 2016, at 06:19 AM, Stephen Powis wrote:
> Oooh I'd love this as well! I really dig the ease of the metric
> framework in storm and have all the metrics go thru one centralized
> config. But as the number of storm hosts and number of tasks grow,
> I've found that Graphite/Grafana has a hard time collecting up all the
> relevant metrics across a lot of wildcarded keys for things like
> hostnames and taskIds to properly display my graphs.
>
> On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway
> <ke...@gmail.com> wrote:
>> One thing I would like to see added (if not already present) is the
>> ability to register metrics that are not tied to a component.
>>
>> As of now, the only non-component metrics are reported by the
>> SystemBolt pseudo-component which feels like a work-around. It
>> reports JVM level metrics like GC time, heap size and other things
>> that aren't associated with a given component.
>>
>> It would be great if application developers could expose similar
>> metrics like this for things like connection pools and other JVM wide
>> objects that aren't unique to a specific component.
>>
>> I don't think this is possible now, is it?
>>
>> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim
>> <ka...@gmail.com> wrote:
>>> Let me start sharing my thought. :)
>>>
>>> 1. Need to enrich docs about metrics / stats.
>>>
>>> In fact, I couldn't see the fact - topology stats are sampled by
>>> default and sample rate is 0.05 - from the docs when I was newbie of
>>> Apache Storm. It made me misleading and made me saying "Why there're
>>> difference between the counts?". I also saw some mails from user@
>>> about same question. If we include this to guide doc that would be
>>> better.
>>>
>>> And Metrics document page[1] seems not well written. I think it has
>>> appropriate headings but lacks contents on each heading.
>>> It should be addressed, and introducing some external metrics
>>> consumer plugins (like storm-graphite[2] from Verisign) would be
>>> great, too.
>>>
>>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>>
>>> Let's postpone considering performance hit at this time.
>>> Ideally, we expect precision of metrics gets better when we increase
>>> sample rate. It affects non-gauge kinds of metrics which are
>>> counter, and latency, and so on.
>>>
>>> Btw, I would like to hear about opinions on latency since I'm not an
>>> expert.
>>> Storm provides only average latency and it's indeed based on sample
>>> rate. Do we feel OK with this? If not how much having also
>>> percentiles can help us?
>>>
>>> Thanks,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>>> Hi Storm users,
>>>>
>>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>>>
>>>> If you subscribed dev@ mailing list, you may have seen that
>>>> recently we're addressing the metrics feature on Apache Storm.
>>>>
>>>> For now, improvements are going forward based on current metrics
>>>> feature.
>>>>
>>>> - Improve (Topology) MetricsConsumer[3]
>>>> - Provide topology metrics in detail (metrics per each stream)[4]
>>>> - (WIP) Introduce Cluster Metrics Consumer
>>>>
>>>> As I don't maintain large cluster for myself, I really want to
>>>> collect the any ideas for improving, any inconveniences, use cases
>>>> of Metrics with community members, so we're on the right way to go
>>>> forward.
>>>>
>>>> Let's talk!
>>>>
>>>> Thanks in advance,
>>>> Jungtaek Lim (HeartSaVioR)
>>
>>
>>
>> --
>> Kevin Conaway http://www.linkedin.com/pub/kevin-conaway/7/107/580/
>> https://github.com/kevinconaway
>>
Links:
1. http://storm.apache.org/releases/1.0.0/Metrics.html
2. https://github.com/verisign/storm-graphite
3. https://issues.apache.org/jira/browse/STORM-1699
4. https://issues.apache.org/jira/browse/STORM-1719
Re: [DISCUSS] Would like to make collective intelligence about
Metrics on Storm
Posted by Stephen Powis <sp...@salesforce.com>.
Oooh I'd love this as well! I really dig the ease of the metric framework
in storm and have all the metrics go thru one centralized config. But as
the number of storm hosts and number of tasks grow, I've found that
Graphite/Grafana has a hard time collecting up all the relevant metrics
across a lot of wildcarded keys for things like hostnames and taskIds to
properly display my graphs.
On Sun, May 1, 2016 at 8:17 AM, Kevin Conaway <ke...@gmail.com>
wrote:
> One thing I would like to see added (if not already present) is the
> ability to register metrics that are not tied to a component.
>
> As of now, the only non-component metrics are reported by the SystemBolt
> pseudo-component which feels like a work-around. It reports JVM level
> metrics like GC time, heap size and other things that aren't associated
> with a given component.
>
> It would be great if application developers could expose similar metrics
> like this for things like connection pools and other JVM wide objects that
> aren't unique to a specific component.
>
> I don't think this is possible now, is it?
>
> On Wed, Apr 20, 2016 at 12:29 AM, Jungtaek Lim <ka...@gmail.com> wrote:
>
>> Let me start sharing my thought. :)
>>
>> 1. Need to enrich docs about metrics / stats.
>>
>> In fact, I couldn't see the fact - topology stats are sampled by default
>> and sample rate is 0.05 - from the docs when I was newbie of Apache
>> Storm. It made me misleading and made me saying "Why there're difference
>> between the counts?". I also saw some mails from user@ about same question.
>> If we include this to guide doc that would be better.
>>
>> And Metrics document page
>> <http://storm.apache.org/releases/1.0.0/Metrics.html> seems not well
>> written. I think it has appropriate headings but lacks contents on each
>> heading.
>> It should be addressed, and introducing some external metrics consumer
>> plugins (like storm-graphite <https://github.com/verisign/storm-graphite>
>> from Verisign) would be great, too.
>>
>> 2. Need to increase sample rate or (ideally) no sampling at all.
>>
>> Let's postpone considering performance hit at this time.
>> Ideally, we expect precision of metrics gets better when we increase
>> sample rate. It affects non-gauge kinds of metrics which are counter,
>> and latency, and so on.
>>
>> Btw, I would like to hear about opinions on latency since I'm not an
>> expert.
>> Storm provides only average latency and it's indeed based on sample rate.
>> Do we feel OK with this? If not how much having also percentiles can help
>> us?
>>
>> Thanks,
>> Jungtaek Lim (HeartSaVioR)
>>
>> 2016년 4월 20일 (수) 오전 10:55, Jungtaek Lim <ka...@gmail.com>님이 작성:
>>
>>> Hi Storm users,
>>>
>>> I'm Jungtaek Lim, committer and PMC member of Apache Storm.
>>>
>>> If you subscribed dev@ mailing list, you may have seen that recently
>>> we're addressing the metrics feature on Apache Storm.
>>>
>>> For now, improvements are going forward based on current metrics feature.
>>>
>>> - Improve (Topology) MetricsConsumer
>>> <https://issues.apache.org/jira/browse/STORM-1699>
>>> - Provide topology metrics in detail (metrics per each stream)
>>> <https://issues.apache.org/jira/browse/STORM-1719>
>>> - (WIP) Introduce Cluster Metrics Consumer
>>>
>>> As I don't maintain large cluster for myself, I really want to collect
>>> the any ideas for improving, any inconveniences, use cases of Metrics with
>>> community members, so we're on the right way to go forward.
>>>
>>> Let's talk!
>>>
>>> Thanks in advance,
>>> Jungtaek Lim (HeartSaVioR)
>>>
>>
>
>
> --
> Kevin Conaway
> http://www.linkedin.com/pub/kevin-conaway/7/107/580/
> https://github.com/kevinconaway
>