You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@storm.apache.org by Tech Id <te...@gmail.com> on 2017/02/08 03:57:09 UTC
Re: Are storm metrics reported through JMX too?
Hi Alessandro, Taylor,
Any more updates on this one?
This seems like a very good feature to have.
Thanks
TI
On Tue, Dec 6, 2016 at 10:37 AM, Alessandro Bellina <
abellina@yahoo-inc.com.invalid> wrote:
> Hi S G,
> Not something I am working on now. What I pushed was just reporter config
> for Taylor since he needed it. I do think that if default reporting works
> we could just go to the rocksdb store and ask it for the metrics there?
> There are other parts of this I haven't published yet but looking to do so
> either this or next week.
> Thanks,
> Alessandro
>
>
>
> On Tuesday, December 6, 2016, 11:10:57 AM CST, S G <
> sg.online.email@gmail.com> wrote:Hey Alessandro,
>
> Thanks for sharing.
> Please share if we have plans to use the metrics-servlets from dropwizard?
> (http://metrics.dropwizard.io/3.1.0/manual/servlets/#adminservlet)
>
> I think it will be very convenient to have the metrics reported through a
> REST API from every process (worker, supervisor, nimbus etc.) and in line
> with most other software like Solr, ES etc. This can be achieved very
> easily if we use the above metric servlets and they also provide ping,
> health-check, thread dump etc which are useful too.
>
> Please ignore if its already there in the code shared above and I could not
> find it.
>
> Thanks,
> SG
>
>
>
> On Mon, Dec 5, 2016 at 9:49 PM, Alessandro Bellina <abellina@yahoo-inc.com
> >
> wrote:
>
> > Hi Taylor
> >
> > Please see latest commit in: https://github.com/abellina/
> > storm/tree/reporters
> >
> > Specifically inside: storm-core/src/jvm/org/apache/storm/metrics2
> >
> > I have a default config that sets up a couple of reporters in
> > default.yaml. The format is inline with what we discussed, but updated
> with
> > what I suggested over the weekend.
> >
> > This is definitely a work in progress, but you should be able to
> > instantiate reporters for your purposes.
> >
> > Thanks,
> >
> > Alessandro
> >
> >
> >
> > On Monday, December 5, 2016, 1:26:41 PM CST, Alessandro Bellina <
> > abellina@yahoo-inc.com.INVALID> wrote:
> > Yes. Will PR this tonight Taylor. Thanks!
> > On Monday, December 5, 2016, 12:37:18 PM CST, P. Taylor Goetz <
> > ptgoetz@gmail.com> wrote:Alessandro,
> >
> > Are you in a position to open a pull request against the metrics_v2
> > branch? I’d like to start integrating the work I’ve been doing with the
> > reporter configuration stuff you have. If what you have is
> incomplete/WIP,
> > that’s not a big deal as the metrics_v2 branch is a feature branch and
> > we’ll have plenty of opportunities to clean things up.
> >
> > -Taylor
> >
> > > On Nov 29, 2016, at 3:27 PM, Alessandro Bellina <
> abellina@yahoo-inc.com>
> > wrote:
> > >
> > > Taylor,
> > >
> > > Ok maybe there is some effort duplication. For the config, I have the
> > bare minimum to get the default reporter up. I'll focus on that since you
> > could use it. Will update JIRA with more.
> > >
> > > Alessandro
> > >
> > >
> > > ----- Forwarded Message -----
> > > From: P. Taylor Goetz <pt...@gmail.com>
> > > To: "dev@storm.apache.org" <de...@storm.apache.org>
> > > Cc: S G <sg...@gmail.com>; naren@narendasan.com <
> > naren@narendasan.com>; Austin Chung <ac...@illinois.edu>
> > > Sent: Tuesday, November 29, 2016, 1:27:58 PM CST
> > > Subject: Re: Are storm metrics reported through JMX too?
> > >
> > > Alessandro,
> > >
> > > Where do you stand with the reporter configuration via the storm.yaml
> > config file?
> > >
> > > I have metrics collection for workers and disruptor queues almost
> ready,
> > but now I’m looking for flexible configuration (right now I have
> reporters
> > hard coded).
> > >
> > > -Taylor
> > >
> > > > On Nov 29, 2016, at 1:57 PM, Alessandro Bellina <
> > abellina@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>>
> > wrote:
> > > >
> > > > S G,
> > > > I am slowly making progress and plan to share something this week,
> > specifically for hooking up a default codahale reporter to the workers.
> > Also the U of I students for the open source class have made progress on
> > their rocksdb implementation for the default store (the stuff I am doing
> > should feed into their store). I don't think any of this is going to be
> in
> > 1.1.0 given that it hasn't really been looked at by others, let alone
> > reviewed.
> > > > Thanks
> > > > Alessandro
> > > > On Monday, November 28, 2016, 4:51:34 PM CST, S G <
> > sg.online.email@gmail.com <ma...@gmail.com>> wrote:Hi,
> > > >
> > > > I see https://issues.apache.org/jira/browse/STORM-2153 <
> > https://issues.apache.org/jira/browse/STORM-2153>being open for this
> > > > and a lot of good discussion happening too.
> > > > There is also a talk for releasing 1.1.0 version soon.
> > > >
> > > > So I just wanted to check if any metric related improvements will be
> > > > available in 1.1.0.
> > > > It would be great to try out these new improvements.
> > > >
> > > > Thanks
> > > > SG
> > > >
> > > > On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <adn5327@gmail.com
> > <ma...@gmail.com>> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'm Abhishek Nigam, one of the students from the group at the
> > University of
> > > >> Illinois; I'm really looking forward to working on this issue! We'll
> > be
> > > >> sure to keep you all updated as to how we progress.
> > > >>
> > > >> Cheers,
> > > >> --
> > > >> Abhishek
> > > >>
> > > >> On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <
> > abellina@yahoo-inc.com <ma...@yahoo-inc.com>
> > > >>>
> > > >> wrote:
> > > >>
> > > >>> + Naren, Austin, and Abhishek, the students from the University of
> > > >>> Illinois Open Source class.
> > > >>>
> > > >>>
> > > >>> On Tuesday, October 11, 2016 11:48 PM, S G <
> > sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>> wrote:
> > > >>>
> > > >>>
> > > >>> Response on this important issue is pretty good. I am happily
> > surprised
> > > >> :)
> > > >>>
> > > >>> I want to mention our strategy for extracting metrics from other
> > > >> products.
> > > >>> We use jolokia_proxy (https://jolokia.org/features/proxy.html <
> > https://jolokia.org/features/proxy.html>) to get
> > > >> JMX
> > > >>> beans from several softwares and feed them to telegraf. That way,
> we
> > > >> avoid
> > > >>> writing custom processors for all these different products.
> > > >>>
> > > >>> Telegraf is quickly becoming a standard for metrics data. (Just see
> > the
> > > >>> list of input plugins here:
> > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/inputs
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins/inputs>). And
> > > >> it
> > > >>> integrates well with several outputs too (
> > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/outputs
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins/outputs>).
> > > >>>
> > > >>> Also since the metrics in JMX, they can be queried by jolokia-agent
> > > >>> installed per node. This avoids the extra metrics-consumer bolt
> > which can
> > > >>> hit the topology throughtput too.
> > > >>>
> > > >>> So I cast my vote in favor of JMX-implementation of metrics.
> > > >>> Other approaches are welcome to be discussed.
> > > >>>
> > > >>> On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina <
> > > >>> abellina@yahoo-inc.com.invalid <mailto:abellina@yahoo-inc.
> > com.invalid>> wrote:
> > > >>>
> > > >>>> blockquote, div.yahoo_quoted { margin-left: 0 !important;
> > > >>> border-left:1px
> > > >>>> #715FFA solid !important; padding-left:1ex !important;
> > > >>>> background-color:white !important; } Yeap that's a requirement
> from
> > our
> > > >>>> perspective (working through this list).
> > > >>>> Sure I think as usual we can start with master with an eye for
> what
> > > >> would
> > > >>>> need to be back ported.
> > > >>>> Sent from Yahoo Mail for iPhone
> > > >>>>
> > > >>>>
> > > >>>> On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz <
> > > >>> ptgoetz@gmail.com <ma...@gmail.com>>
> > > >>>> wrote:
> > > >>>>
> > > >>>> I hope I didn't come across as overly critical. You did the best
> > with
> > > >>> what
> > > >>>> you had to work with. Which isn't pretty.
> > > >>>>
> > > >>>> We could potentially do a parallel metrics API in 1.1, 1.2, or
> > master
> > > >> and
> > > >>>> still stay close to semantic versioning...?
> > > >>>>
> > > >>>> -Taylor
> > > >>>>
> > > >>>>> On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <kabhwan@gmail.com
> > <ma...@gmail.com>> wrote:
> > > >>>>>
> > > >>>>> Yeah I admit that configuration flag was bad for me also, but I
> > have
> > > >> no
> > > >>>>> alternatives. Only way to avoid struggling with design limitation
> > is
> > > >>>> revamp
> > > >>>>> / redesign.
> > > >>>>> Thanks S G for exposing willingness of volunteer and great news
> > > >>>> Alessandro
> > > >>>>> for that project.
> > > >>>>> Alessandro, could you forward the upcoming news for the project
> to
> > > >> dev@
> > > >>>>> list?
> > > >>>>>
> > > >>>>> - Jungtaek Lim (HeartSaVioR)
> > > >>>>>
> > > >>>>> 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <ptgoetz@gmail.com
> > <ma...@gmail.com>>님이
> > > >> 작성:
> > > >>>>>
> > > >>>>>> I was thinking on a smaller scale in terms of effort, but the
> > more I
> > > >>>> think
> > > >>>>>> about it, the more supportive I would be of a full revamp (new
> > API)
> > > >>> for
> > > >>>>>> metrics based on Coda Hale's metrics library. It's proven and
> > > >> stable.
> > > >>>> I've
> > > >>>>>> used it many times. I think either approach would be roughly the
> > > >> same
> > > >>>>>> amount of work.
> > > >>>>>>
> > > >>>>>> Some of the metrics API improvements in the 1.1.x branch are
> nice,
> > > >> but
> > > >>>>>> IMHO are lipstick on a pig.
> > > >>>>>>
> > > >>>>>> With apologies to Jungtaek, who has done amazing work all across
> > the
> > > >>>>>> codebase, I'm a little squeamish about the proposed change to
> > > >> metrics
> > > >>>> that
> > > >>>>>> changes the consumer API based on a configuration flag (don't
> know
> > > >> the
> > > >>>> PR
> > > >>>>>> number offhand).
> > > >>>>>>
> > > >>>>>> I'm +1 for moving in this direction (revamped metrics). Let's
> end
> > > >> the
> > > >>>>>> metrics pain.
> > > >>>>>>
> > > >>>>>> -Taylor
> > > >>>>>>
> > > >>>>>>> On Oct 11, 2016, at 10:15 AM, Bobby Evans <
> > > >>> evans@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I agree that IMetricsConsumer is not good, but the reality is
> > that
> > > >>> all
> > > >>>>>> of the metrics system needs to be redone. The problem is that
> we
> > > >> ship
> > > >>>> an
> > > >>>>>> object as a metric. If I get an object I have no idea what it
> is
> > > >> hand
> > > >>>>>> hence no idea how to report it or what to do with it. What is
> > more
> > > >>> the
> > > >>>>>> common types we use in the metrics we provide are really not
> > enough.
> > > >>>> For
> > > >>>>>> example CountMetric sends a Long. Well when I get it in the
> > metrics
> > > >>>>>> consumer I have no idea if I should report it like a counter or
> > if I
> > > >>>> should
> > > >>>>>> report it like a gauge (something that every metrics system I
> have
> > > >>> used
> > > >>>>>> wants to know). But then we support pre-aggregation of the
> > metrics
> > > >>> with
> > > >>>>>> IReducer so the number I get might be an average instead of
> > either a
> > > >>>> gauge
> > > >>>>>> or a counter, which no good metrics system will want to collect
> > > >>> because
> > > >>>> I
> > > >>>>>> cannot aggregate it with anything else, the math just does not
> > work.
> > > >>>>>>> The proposal I have said before and I still believe is that we
> > need
> > > >>> to
> > > >>>>>> put in place a parallel metrics API/system. We will deprecate
> all
> > > >> of
> > > >>>>>> https://git.corp.yahoo.com/storm/storm/tree/master- <
> > https://git.corp.yahoo.com/storm/storm/tree/master->
> > > >>>> security/storm-core/src/jvm/backtype/storm/metric/api
> > > >>>>>> and create a new parallel one that provides an API similar to
> > > >>>>>> http://metrics.dropwizard.io/3.1.0/.
> > > >>> <http://metrics.dropwizard.io/3.1.0/ <
> http://metrics.dropwizard.io/
> > 3.1.0/>> I would even be fine in just
> > > >>>> using
> > > >>>>>> their API and exposing that to end users. Dropwizard has solved
> > all
> > > >>> of
> > > >>>>>> these problems already and I don't see a reason to reinvent the
> > > >> wheel.
> > > >>>> I
> > > >>>>>> don't personally see a lot of value in trying to send all of the
> > > >>> metrics
> > > >>>>>> through storm itslef. I am fine if we are able to support that,
> > but
> > > >>> it
> > > >>>> is
> > > >>>>>> far from a requirement. - Bobby
> > > >>>>>>>
> > > >>>>>>> On Monday, October 10, 2016 10:47 PM, S G <
> > > >>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> +1
> > > >>>>>>> We can probably start by opening a JIRA for this and adding a
> > > >> design
> > > >>>>>>> approach for the same?
> > > >>>>>>> I would like to help in the coding-effort for this.
> > > >>>>>>>
> > > >>>>>>> -SG
> > > >>>>>>>
> > > >>>>>>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <
> > > >> ptgoetz@gmail.com <ma...@gmail.com>
> > > >>>>
> > > >>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> I’ve been thinking about metrics lately, specifically the fact
> > > >> that
> > > >>>>>> people
> > > >>>>>>>> tend to struggle with implementing a metrics consumer. (Like
> > this
> > > >>> one
> > > >>>>>> [1]).
> > > >>>>>>>>
> > > >>>>>>>> The IMetricsConsumer interface is pretty low level, and common
> > > >>>>>>>> aggregations, calculations, etc. are left up to each
> individual
> > > >>>>>>>> implementation. That seems like an area where further
> > abstraction
> > > >>>> would
> > > >>>>>>>> make it easier to support different back ends (Graphite, JMX,
> > > >>> Splunk,
> > > >>>>>> etc.).
> > > >>>>>>>>
> > > >>>>>>>> My thought is to create an abstract IMetricsConsumer
> > > >> implementation
> > > >>>> that
> > > >>>>>>>> does common aggregations and calculations, and then delegates
> > to a
> > > >>>>>> plugable
> > > >>>>>>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.).
> That
> > > >>> would
> > > >>>>>>>> greatly simplify the effort required to integrate with various
> > > >>>> external
> > > >>>>>>>> metrics systems. I know of at least a few users that would be
> > > >>>>>> interested,
> > > >>>>>>>> one is currently scraping the logs from LoggingMetricsConsumer
> > and
> > > >>>>>> polling
> > > >>>>>>>> the Storm REST API for their metrics.
> > > >>>>>>>>
> > > >>>>>>>> -Taylor
> > > >>>>>>>>
> > > >>>>>>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ <
> > http://twocentsonsoftware.blogspot.co.il/2014/12/>
> > > >>>>>>>> sending-out-storm-metrics.html
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans
> > > >>>> <evans@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> First of all the server exposes essentially the same
> interface
> > > >> that
> > > >>>> the
> > > >>>>>>>> IMetricsConsumer exposes. It mostly just adds a bunch of
> > overhead
> > > >>> in
> > > >>>>>> the
> > > >>>>>>>> middle to serialize out the objects send them over http to
> > another
> > > >>>>>> process
> > > >>>>>>>> which then has to deserialize them and process them. If you
> > > >> really
> > > >>>>>> don't
> > > >>>>>>>> need the metrics to show up on a special known box you can
> have
> > > >> that
> > > >>>>>> exact
> > > >>>>>>>> same code running inside the metrics consumer without all of
> the
> > > >>>>>> overhead.
> > > >>>>>>>>> The server/client are insecure, have to deal with thread
> issues
> > > >>> that
> > > >>>> a
> > > >>>>>>>> normal IMetricsConsumer does not, and are not written to be
> > robust
> > > >>> (If
> > > >>>>>> the
> > > >>>>>>>> HTTP server is down the consumer crashes and continues to
> crash
> > > >>> until
> > > >>>>>> the
> > > >>>>>>>> server is brought back up). It was written very quickly for a
> > > >> test
> > > >>>>>>>> situation and it honestly never crossed my mind that anyone
> > would
> > > >>> want
> > > >>>>>> to
> > > >>>>>>>> use it in production.
> > > >>>>>>>>>
> > > >>>>>>>>> - Bobby
> > > >>>>>>>>>
> > > >>>>>>>>> On Monday, October 10, 2016 10:59 AM, S G <
> > > >>>>>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks Bobby.
> > > >>>>>>>>>
> > > >>>>>>>>> If we write our own metrics consumer, how do we ensure that
> it
> > is
> > > >>>>>> better
> > > >>>>>>>>> than HttpForwardingMetricsServer? In other words, what
> aspects
> > of
> > > >>> the
> > > >>>>>>>>> HttpForwardingMetricsServer
> > > >>>>>>>>> should we avoid to make our own metrics consumer better and
> > ready
> > > >>> for
> > > >>>>>>>>> production?
> > > >>>>>>>>>
> > > >>>>>>>>> Is versign/storm-graphite <https://github.com/verisign/ <
> > https://github.com/verisign/>
> > > >>>> storm-graphite>
> > > >>>>>>>>> production
> > > >>>>>>>>> ready?
> > > >>>>>>>>>
> > > >>>>>>>>> Also, we should add a line about production-readiness of
> > > >>>>>>>>> HttpForwardingMetricsServer
> > > >>>>>>>>> in the documentation at http://storm.apache.org/ <
> > http://storm.apache.org/>
> > > >>>>>>>> releases/1.0.2/Metrics.html
> > > >>>>>>>>> (We were just about to think seriously on using this for
> > > >> production
> > > >>>> as
> > > >>>>>> we
> > > >>>>>>>>> thought this to be the standard solution for metrics'
> > consumption
> > > >>> in
> > > >>>>>> 1.0+
> > > >>>>>>>>> version).
> > > >>>>>>>>>
> > > >>>>>>>>> -SG
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <
> > > >> evans@yahoo-inc.com <ma...@yahoo-inc.com>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> First of all there really are two different sets of metrics.
> > > >> One
> > > >>>> set
> > > >>>>>> is
> > > >>>>>>>>>> the topology metrics and the other set is the daemon metrics
> > > >>>> (metrics
> > > >>>>>>>> for
> > > >>>>>>>>>> things like the ui and nimbus). The JmxPreparableReporter
> > > >> plugin
> > > >>>> only
> > > >>>>>>>>>> exposes daemon metrics not the topology metrics through JMX.
> > > >>>> Exposing
> > > >>>>>>>>>> topology metrics through JMX is a non trivial task. The
> > current
> > > >>>>>> metrics
> > > >>>>>>>>>> feature was not designed for this. We are in the process of
> > > >>> trying
> > > >>>> to
> > > >>>>>>>>>> redesign the metrics system to allow for features like this,
> > but
> > > >>> it
> > > >>>> is
> > > >>>>>>>>>> still a ways off.
> > > >>>>>>>>>>
> > > >>>>>>>>>> - Bobby
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Saturday, October 8, 2016 11:39 AM, S G <
> > > >>>> sg.online.email@gmail.com <ma...@gmail.com>
> > > >>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks Bobby,
> > > >>>>>>>>>>
> > > >>>>>>>>>> We will need some kind of IMetricsConsumer to talk to
> > telegraf.
> > > >>>>>>>>>> Many other softwares like Solr, Elastic-Search, Cassandra
> etc.
> > > >>>> provide
> > > >>>>>>>>>> metrics through a URL making it very easy to consume by
> tools
> > > >> like
> > > >>>>>>>> telegraf.
> > > >>>>>>>>>> How about a IMetricsConsumer that will run on storm-ui and
> > > >> provide
> > > >>>> the
> > > >>>>>>>>>> metrics through a URL such as <storm-ui-host>/metrics ?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Also, I see the following option in defaults.yaml:
> > > >>>>>>>>>> #default storm daemon metrics reporter plugins
> > > >>>>>>>>>> storm.daemon.metrics.reporter.plugins:
> > > >>>>>>>>>> - "org.apache.storm.daemon.metrics.reporters.
> > > >>>>>>>> JmxPreparableReporter"
> > > >>>>>>>>>>
> > > >>>>>>>>>> Is this a good option to use for converting metrics into
> JMX ?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks
> > > >>>>>>>>>> SG
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans
> > > >>>>>> <evans@yahoo-inc.com.invalid <mailto:evans@yahoo-inc.com.
> invalid>
> > > >>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> HttpForwardingMetricsServer is a real hack intended really
> for
> > > >>>>>> tests. I
> > > >>>>>>>>>> know I wrote it :). Please don't use it in production. You
> > can
> > > >>>> write
> > > >>>>>>>> your
> > > >>>>>>>>>> own IMetricesConsumer to do whatever you want to with the
> > > >> metrics.
> > > >>>>>>>>>> https://github.com/apache/ <https://github.com/apache/>
> > storm/blob/master/storm-core/
> > > >>>>>>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java
> > > >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/ <
> > https://github.com/apache/storm/blob/master/storm-core/>
> > > >>>>>>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java>
> > > >>>>>>>>>>
> > > >>>>>>>>>> That is the correct way to get the data out. If you want to
> > > >>> write a
> > > >>>>>>>>>> bridge to JMX for this that might work, but going directly
> to
> > > >>>>>> telegraph
> > > >>>>>>>>>> would probably be better. - Bobby
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thursday, October 6, 2016 1:43 PM, S G <
> > > >>>>>>>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi,
> > > >>>>>>>>>>
> > > >>>>>>>>>> We want to use Telegraf (
> > > >>>>>>>>>> https://github.com/influxdata/ <https://github.com/
> > influxdata/>telegraf/tree/master/plugins
> > > >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins>>)
> > > >> for
> > > >>>>>>>> getting
> > > >>>>>>>>>> storm's metrics.
> > > >>>>>>>>>>
> > > >>>>>>>>>> But we do not want to add a HttpForwardingMetricsServer just
> > to
> > > >>> get
> > > >>>>>> the
> > > >>>>>>>>>> metrics and send them to telegraf.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Other option is to use Jolokia (https://jolokia.org/ <
> > https://jolokia.org/>) that can
> > > >>> read
> > > >>>>>> JMX
> > > >>>>>>>>>> and
> > > >>>>>>>>>> write into telegraf.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Does storm report all its metrics (including those of custom
> > > >>>>>>>> spouts/bolts)
> > > >>>>>>>>>> into JMX?
> > > >>>>>>>>>> Or spawning a HttpForwardingMetricsServer is the only
> option?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks
> > > >>>>>>>>>> SG
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> >
>
Re: Are storm metrics reported through JMX too?
Posted by Erik Weathers <ew...@groupon.com.INVALID>.
Also this proposal would be a real problem for the Storm-on-Mesos project.
The ports would have to be dynamically allocated from Mesos in such a
scenario, and we'd need some way to register and retrieve these ports in
order to drive whatever is then pulling the metrics out from them. The
same problem exists for the logviewer, as I've documented here:
- https://issues.apache.org/jira/browse/STORM-1342
On Wed, Feb 8, 2017 at 7:16 AM, Bobby Evans <ev...@yahoo-inc.com.invalid>
wrote:
> For me the big issue is that not every process has a web server on it.
> That may change in the future but for now only the ui and the logviewer
> have a web server up and running. What is more if we wanted to do this for
> the workers we would need to think about an alternative port that would be
> free for a web server to be on, the resources a web server would use inside
> the worker and how would authentication work to those worker processes.
> SPNEGO would be really difficult to make work.
>
>
> - Bobby
>
> On Tuesday, February 7, 2017, 9:57:27 PM CST, Tech Id <
> tech.login.id2@gmail.com> wrote:Hi Alessandro, Taylor,
>
> Any more updates on this one?
> This seems like a very good feature to have.
>
> Thanks
> TI
>
> On Tue, Dec 6, 2016 at 10:37 AM, Alessandro Bellina <
> abellina@yahoo-inc.com.invalid> wrote:
>
> > Hi S G,
> > Not something I am working on now. What I pushed was just reporter config
> > for Taylor since he needed it. I do think that if default reporting works
> > we could just go to the rocksdb store and ask it for the metrics there?
> > There are other parts of this I haven't published yet but looking to do
> so
> > either this or next week.
> > Thanks,
> > Alessandro
> >
> >
> >
> > On Tuesday, December 6, 2016, 11:10:57 AM CST, S G <
> > sg.online.email@gmail.com> wrote:Hey Alessandro,
> >
> > Thanks for sharing.
> > Please share if we have plans to use the metrics-servlets from
> dropwizard?
> > (http://metrics.dropwizard.io/3.1.0/manual/servlets/#adminservlet)
> >
> > I think it will be very convenient to have the metrics reported through a
> > REST API from every process (worker, supervisor, nimbus etc.) and in line
> > with most other software like Solr, ES etc. This can be achieved very
> > easily if we use the above metric servlets and they also provide ping,
> > health-check, thread dump etc which are useful too.
> >
> > Please ignore if its already there in the code shared above and I could
> not
> > find it.
> >
> > Thanks,
> > SG
> >
> >
> >
> > On Mon, Dec 5, 2016 at 9:49 PM, Alessandro Bellina <
> abellina@yahoo-inc.com
> > >
> > wrote:
> >
> > > Hi Taylor
> > >
> > > Please see latest commit in: https://github.com/abellina/
> > > storm/tree/reporters
> > >
> > > Specifically inside: storm-core/src/jvm/org/apache/storm/metrics2
> > >
> > > I have a default config that sets up a couple of reporters in
> > > default.yaml. The format is inline with what we discussed, but updated
> > with
> > > what I suggested over the weekend.
> > >
> > > This is definitely a work in progress, but you should be able to
> > > instantiate reporters for your purposes.
> > >
> > > Thanks,
> > >
> > > Alessandro
> > >
> > >
> > >
> > > On Monday, December 5, 2016, 1:26:41 PM CST, Alessandro Bellina <
> > > abellina@yahoo-inc.com.INVALID> wrote:
> > > Yes. Will PR this tonight Taylor. Thanks!
> > > On Monday, December 5, 2016, 12:37:18 PM CST, P. Taylor Goetz <
> > > ptgoetz@gmail.com> wrote:Alessandro,
> > >
> > > Are you in a position to open a pull request against the metrics_v2
> > > branch? I’d like to start integrating the work I’ve been doing with the
> > > reporter configuration stuff you have. If what you have is
> > incomplete/WIP,
> > > that’s not a big deal as the metrics_v2 branch is a feature branch and
> > > we’ll have plenty of opportunities to clean things up.
> > >
> > > -Taylor
> > >
> > > > On Nov 29, 2016, at 3:27 PM, Alessandro Bellina <
> > abellina@yahoo-inc.com>
> > > wrote:
> > > >
> > > > Taylor,
> > > >
> > > > Ok maybe there is some effort duplication. For the config, I have the
> > > bare minimum to get the default reporter up. I'll focus on that since
> you
> > > could use it. Will update JIRA with more.
> > > >
> > > > Alessandro
> > > >
> > > >
> > > > ----- Forwarded Message -----
> > > > From: P. Taylor Goetz <pt...@gmail.com>
> > > > To: "dev@storm.apache.org" <de...@storm.apache.org>
> > > > Cc: S G <sg...@gmail.com>; naren@narendasan.com <
> > > naren@narendasan.com>; Austin Chung <ac...@illinois.edu>
> > > > Sent: Tuesday, November 29, 2016, 1:27:58 PM CST
> > > > Subject: Re: Are storm metrics reported through JMX too?
> > > >
> > > > Alessandro,
> > > >
> > > > Where do you stand with the reporter configuration via the storm.yaml
> > > config file?
> > > >
> > > > I have metrics collection for workers and disruptor queues almost
> > ready,
> > > but now I’m looking for flexible configuration (right now I have
> > reporters
> > > hard coded).
> > > >
> > > > -Taylor
> > > >
> > > > > On Nov 29, 2016, at 1:57 PM, Alessandro Bellina <
> > > abellina@yahoo-inc.com.INVALID <mailto:abellina@yahoo-inc.com.INVALID
> >>
> > > wrote:
> > > > >
> > > > > S G,
> > > > > I am slowly making progress and plan to share something this week,
> > > specifically for hooking up a default codahale reporter to the workers.
> > > Also the U of I students for the open source class have made progress
> on
> > > their rocksdb implementation for the default store (the stuff I am
> doing
> > > should feed into their store). I don't think any of this is going to be
> > in
> > > 1.1.0 given that it hasn't really been looked at by others, let alone
> > > reviewed.
> > > > > Thanks
> > > > > Alessandro
> > > > > On Monday, November 28, 2016, 4:51:34 PM CST, S G <
> > > sg.online.email@gmail.com <ma...@gmail.com>>
> wrote:Hi,
> > > > >
> > > > > I see https://issues.apache.org/jira/browse/STORM-2153 <
> > > https://issues.apache.org/jira/browse/STORM-2153>being open for this
> > > > > and a lot of good discussion happening too.
> > > > > There is also a talk for releasing 1.1.0 version soon.
> > > > >
> > > > > So I just wanted to check if any metric related improvements will
> be
> > > > > available in 1.1.0.
> > > > > It would be great to try out these new improvements.
> > > > >
> > > > > Thanks
> > > > > SG
> > > > >
> > > > > On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <adn5327@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > > >
> > > > >> Hi all,
> > > > >>
> > > > >> I'm Abhishek Nigam, one of the students from the group at the
> > > University of
> > > > >> Illinois; I'm really looking forward to working on this issue!
> We'll
> > > be
> > > > >> sure to keep you all updated as to how we progress.
> > > > >>
> > > > >> Cheers,
> > > > >> --
> > > > >> Abhishek
> > > > >>
> > > > >> On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <
> > > abellina@yahoo-inc.com <ma...@yahoo-inc.com>
> > > > >>>
> > > > >> wrote:
> > > > >>
> > > > >>> + Naren, Austin, and Abhishek, the students from the University
> of
> > > > >>> Illinois Open Source class.
> > > > >>>
> > > > >>>
> > > > >>> On Tuesday, October 11, 2016 11:48 PM, S G <
> > > sg.online.email@gmail.com <ma...@gmail.com>>
> > > > >>> wrote:
> > > > >>>
> > > > >>>
> > > > >>> Response on this important issue is pretty good. I am happily
> > > surprised
> > > > >> :)
> > > > >>>
> > > > >>> I want to mention our strategy for extracting metrics from other
> > > > >> products.
> > > > >>> We use jolokia_proxy (https://jolokia.org/features/proxy.html <
> > > https://jolokia.org/features/proxy.html>) to get
> > > > >> JMX
> > > > >>> beans from several softwares and feed them to telegraf. That way,
> > we
> > > > >> avoid
> > > > >>> writing custom processors for all these different products.
> > > > >>>
> > > > >>> Telegraf is quickly becoming a standard for metrics data. (Just
> see
> > > the
> > > > >>> list of input plugins here:
> > > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/
> inputs
> > <
> > > https://github.com/influxdata/telegraf/tree/master/plugins/inputs>).
> And
> > > > >> it
> > > > >>> integrates well with several outputs too (
> > > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/
> outputs
> > <
> > > https://github.com/influxdata/telegraf/tree/master/plugins/outputs>).
> > > > >>>
> > > > >>> Also since the metrics in JMX, they can be queried by
> jolokia-agent
> > > > >>> installed per node. This avoids the extra metrics-consumer bolt
> > > which can
> > > > >>> hit the topology throughtput too.
> > > > >>>
> > > > >>> So I cast my vote in favor of JMX-implementation of metrics.
> > > > >>> Other approaches are welcome to be discussed.
> > > > >>>
> > > > >>> On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina <
> > > > >>> abellina@yahoo-inc.com.invalid <mailto:abellina@yahoo-inc.
> > > com.invalid>> wrote:
> > > > >>>
> > > > >>>> blockquote, div.yahoo_quoted { margin-left: 0 !important;
> > > > >>> border-left:1px
> > > > >>>> #715FFA solid !important; padding-left:1ex !important;
> > > > >>>> background-color:white !important; } Yeap that's a requirement
> > from
> > > our
> > > > >>>> perspective (working through this list).
> > > > >>>> Sure I think as usual we can start with master with an eye for
> > what
> > > > >> would
> > > > >>>> need to be back ported.
> > > > >>>> Sent from Yahoo Mail for iPhone
> > > > >>>>
> > > > >>>>
> > > > >>>> On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz <
> > > > >>> ptgoetz@gmail.com <ma...@gmail.com>>
> > > > >>>> wrote:
> > > > >>>>
> > > > >>>> I hope I didn't come across as overly critical. You did the best
> > > with
> > > > >>> what
> > > > >>>> you had to work with. Which isn't pretty.
> > > > >>>>
> > > > >>>> We could potentially do a parallel metrics API in 1.1, 1.2, or
> > > master
> > > > >> and
> > > > >>>> still stay close to semantic versioning...?
> > > > >>>>
> > > > >>>> -Taylor
> > > > >>>>
> > > > >>>>> On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <kabhwan@gmail.com
> > > <ma...@gmail.com>> wrote:
> > > > >>>>>
> > > > >>>>> Yeah I admit that configuration flag was bad for me also, but I
> > > have
> > > > >> no
> > > > >>>>> alternatives. Only way to avoid struggling with design
> limitation
> > > is
> > > > >>>> revamp
> > > > >>>>> / redesign.
> > > > >>>>> Thanks S G for exposing willingness of volunteer and great news
> > > > >>>> Alessandro
> > > > >>>>> for that project.
> > > > >>>>> Alessandro, could you forward the upcoming news for the project
> > to
> > > > >> dev@
> > > > >>>>> list?
> > > > >>>>>
> > > > >>>>> - Jungtaek Lim (HeartSaVioR)
> > > > >>>>>
> > > > >>>>> 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <ptgoetz@gmail.com
> > > <ma...@gmail.com>>님이
> > > > >> 작성:
> > > > >>>>>
> > > > >>>>>> I was thinking on a smaller scale in terms of effort, but the
> > > more I
> > > > >>>> think
> > > > >>>>>> about it, the more supportive I would be of a full revamp (new
> > > API)
> > > > >>> for
> > > > >>>>>> metrics based on Coda Hale's metrics library. It's proven and
> > > > >> stable.
> > > > >>>> I've
> > > > >>>>>> used it many times. I think either approach would be roughly
> the
> > > > >> same
> > > > >>>>>> amount of work.
> > > > >>>>>>
> > > > >>>>>> Some of the metrics API improvements in the 1.1.x branch are
> > nice,
> > > > >> but
> > > > >>>>>> IMHO are lipstick on a pig.
> > > > >>>>>>
> > > > >>>>>> With apologies to Jungtaek, who has done amazing work all
> across
> > > the
> > > > >>>>>> codebase, I'm a little squeamish about the proposed change to
> > > > >> metrics
> > > > >>>> that
> > > > >>>>>> changes the consumer API based on a configuration flag (don't
> > know
> > > > >> the
> > > > >>>> PR
> > > > >>>>>> number offhand).
> > > > >>>>>>
> > > > >>>>>> I'm +1 for moving in this direction (revamped metrics). Let's
> > end
> > > > >> the
> > > > >>>>>> metrics pain.
> > > > >>>>>>
> > > > >>>>>> -Taylor
> > > > >>>>>>
> > > > >>>>>>> On Oct 11, 2016, at 10:15 AM, Bobby Evans <
> > > > >>> evans@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>
> > > > >>>>>
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>> I agree that IMetricsConsumer is not good, but the reality is
> > > that
> > > > >>> all
> > > > >>>>>> of the metrics system needs to be redone. The problem is that
> > we
> > > > >> ship
> > > > >>>> an
> > > > >>>>>> object as a metric. If I get an object I have no idea what it
> > is
> > > > >> hand
> > > > >>>>>> hence no idea how to report it or what to do with it. What is
> > > more
> > > > >>> the
> > > > >>>>>> common types we use in the metrics we provide are really not
> > > enough.
> > > > >>>> For
> > > > >>>>>> example CountMetric sends a Long. Well when I get it in the
> > > metrics
> > > > >>>>>> consumer I have no idea if I should report it like a counter
> or
> > > if I
> > > > >>>> should
> > > > >>>>>> report it like a gauge (something that every metrics system I
> > have
> > > > >>> used
> > > > >>>>>> wants to know). But then we support pre-aggregation of the
> > > metrics
> > > > >>> with
> > > > >>>>>> IReducer so the number I get might be an average instead of
> > > either a
> > > > >>>> gauge
> > > > >>>>>> or a counter, which no good metrics system will want to
> collect
> > > > >>> because
> > > > >>>> I
> > > > >>>>>> cannot aggregate it with anything else, the math just does not
> > > work.
> > > > >>>>>>> The proposal I have said before and I still believe is that
> we
> > > need
> > > > >>> to
> > > > >>>>>> put in place a parallel metrics API/system. We will deprecate
> > all
> > > > >> of
> > > > >>>>>> https://git.corp.yahoo.com/storm/storm/tree/master- <
> > > https://git.corp.yahoo.com/storm/storm/tree/master->
> > > > >>>> security/storm-core/src/jvm/backtype/storm/metric/api
> > > > >>>>>> and create a new parallel one that provides an API similar to
> > > > >>>>>> http://metrics.dropwizard.io/3.1.0/.
> > > > >>> <http://metrics.dropwizard.io/3.1.0/ <
> > http://metrics.dropwizard.io/
> > > 3.1.0/>> I would even be fine in just
> > > > >>>> using
> > > > >>>>>> their API and exposing that to end users. Dropwizard has
> solved
> > > all
> > > > >>> of
> > > > >>>>>> these problems already and I don't see a reason to reinvent
> the
> > > > >> wheel.
> > > > >>>> I
> > > > >>>>>> don't personally see a lot of value in trying to send all of
> the
> > > > >>> metrics
> > > > >>>>>> through storm itslef. I am fine if we are able to support
> that,
> > > but
> > > > >>> it
> > > > >>>> is
> > > > >>>>>> far from a requirement. - Bobby
> > > > >>>>>>>
> > > > >>>>>>> On Monday, October 10, 2016 10:47 PM, S G <
> > > > >>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > > >>>>>> wrote:
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> +1
> > > > >>>>>>> We can probably start by opening a JIRA for this and adding a
> > > > >> design
> > > > >>>>>>> approach for the same?
> > > > >>>>>>> I would like to help in the coding-effort for this.
> > > > >>>>>>>
> > > > >>>>>>> -SG
> > > > >>>>>>>
> > > > >>>>>>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <
> > > > >> ptgoetz@gmail.com <ma...@gmail.com>
> > > > >>>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>
> > > > >>>>>>>> I’ve been thinking about metrics lately, specifically the
> fact
> > > > >> that
> > > > >>>>>> people
> > > > >>>>>>>> tend to struggle with implementing a metrics consumer. (Like
> > > this
> > > > >>> one
> > > > >>>>>> [1]).
> > > > >>>>>>>>
> > > > >>>>>>>> The IMetricsConsumer interface is pretty low level, and
> common
> > > > >>>>>>>> aggregations, calculations, etc. are left up to each
> > individual
> > > > >>>>>>>> implementation. That seems like an area where further
> > > abstraction
> > > > >>>> would
> > > > >>>>>>>> make it easier to support different back ends (Graphite,
> JMX,
> > > > >>> Splunk,
> > > > >>>>>> etc.).
> > > > >>>>>>>>
> > > > >>>>>>>> My thought is to create an abstract IMetricsConsumer
> > > > >> implementation
> > > > >>>> that
> > > > >>>>>>>> does common aggregations and calculations, and then
> delegates
> > > to a
> > > > >>>>>> plugable
> > > > >>>>>>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.).
> > That
> > > > >>> would
> > > > >>>>>>>> greatly simplify the effort required to integrate with
> various
> > > > >>>> external
> > > > >>>>>>>> metrics systems. I know of at least a few users that would
> be
> > > > >>>>>> interested,
> > > > >>>>>>>> one is currently scraping the logs from
> LoggingMetricsConsumer
> > > and
> > > > >>>>>> polling
> > > > >>>>>>>> the Storm REST API for their metrics.
> > > > >>>>>>>>
> > > > >>>>>>>> -Taylor
> > > > >>>>>>>>
> > > > >>>>>>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ <
> > > http://twocentsonsoftware.blogspot.co.il/2014/12/>
> > > > >>>>>>>> sending-out-storm-metrics.html
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans
> > > > >>>> <evans@yahoo-inc.com.INVALID <mailto:evans@yahoo-inc.com.
> INVALID>
> > > > >>>>>>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>> First of all the server exposes essentially the same
> > interface
> > > > >> that
> > > > >>>> the
> > > > >>>>>>>> IMetricsConsumer exposes. It mostly just adds a bunch of
> > > overhead
> > > > >>> in
> > > > >>>>>> the
> > > > >>>>>>>> middle to serialize out the objects send them over http to
> > > another
> > > > >>>>>> process
> > > > >>>>>>>> which then has to deserialize them and process them. If you
> > > > >> really
> > > > >>>>>> don't
> > > > >>>>>>>> need the metrics to show up on a special known box you can
> > have
> > > > >> that
> > > > >>>>>> exact
> > > > >>>>>>>> same code running inside the metrics consumer without all of
> > the
> > > > >>>>>> overhead.
> > > > >>>>>>>>> The server/client are insecure, have to deal with thread
> > issues
> > > > >>> that
> > > > >>>> a
> > > > >>>>>>>> normal IMetricsConsumer does not, and are not written to be
> > > robust
> > > > >>> (If
> > > > >>>>>> the
> > > > >>>>>>>> HTTP server is down the consumer crashes and continues to
> > crash
> > > > >>> until
> > > > >>>>>> the
> > > > >>>>>>>> server is brought back up). It was written very quickly
> for a
> > > > >> test
> > > > >>>>>>>> situation and it honestly never crossed my mind that anyone
> > > would
> > > > >>> want
> > > > >>>>>> to
> > > > >>>>>>>> use it in production.
> > > > >>>>>>>>>
> > > > >>>>>>>>> - Bobby
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Monday, October 10, 2016 10:59 AM, S G <
> > > > >>>>>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>> Thanks Bobby.
> > > > >>>>>>>>>
> > > > >>>>>>>>> If we write our own metrics consumer, how do we ensure that
> > it
> > > is
> > > > >>>>>> better
> > > > >>>>>>>>> than HttpForwardingMetricsServer? In other words, what
> > aspects
> > > of
> > > > >>> the
> > > > >>>>>>>>> HttpForwardingMetricsServer
> > > > >>>>>>>>> should we avoid to make our own metrics consumer better and
> > > ready
> > > > >>> for
> > > > >>>>>>>>> production?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Is versign/storm-graphite <https://github.com/verisign/ <
> > > https://github.com/verisign/>
> > > > >>>> storm-graphite>
> > > > >>>>>>>>> production
> > > > >>>>>>>>> ready?
> > > > >>>>>>>>>
> > > > >>>>>>>>> Also, we should add a line about production-readiness of
> > > > >>>>>>>>> HttpForwardingMetricsServer
> > > > >>>>>>>>> in the documentation at http://storm.apache.org/ <
> > > http://storm.apache.org/>
> > > > >>>>>>>> releases/1.0.2/Metrics.html
> > > > >>>>>>>>> (We were just about to think seriously on using this for
> > > > >> production
> > > > >>>> as
> > > > >>>>>> we
> > > > >>>>>>>>> thought this to be the standard solution for metrics'
> > > consumption
> > > > >>> in
> > > > >>>>>> 1.0+
> > > > >>>>>>>>> version).
> > > > >>>>>>>>>
> > > > >>>>>>>>> -SG
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <
> > > > >> evans@yahoo-inc.com <ma...@yahoo-inc.com>>
> > > > >>>>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> First of all there really are two different sets of
> metrics.
> > > > >> One
> > > > >>>> set
> > > > >>>>>> is
> > > > >>>>>>>>>> the topology metrics and the other set is the daemon
> metrics
> > > > >>>> (metrics
> > > > >>>>>>>> for
> > > > >>>>>>>>>> things like the ui and nimbus). The JmxPreparableReporter
> > > > >> plugin
> > > > >>>> only
> > > > >>>>>>>>>> exposes daemon metrics not the topology metrics through
> JMX.
> > > > >>>> Exposing
> > > > >>>>>>>>>> topology metrics through JMX is a non trivial task. The
> > > current
> > > > >>>>>> metrics
> > > > >>>>>>>>>> feature was not designed for this. We are in the process
> of
> > > > >>> trying
> > > > >>>> to
> > > > >>>>>>>>>> redesign the metrics system to allow for features like
> this,
> > > but
> > > > >>> it
> > > > >>>> is
> > > > >>>>>>>>>> still a ways off.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> - Bobby
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Saturday, October 8, 2016 11:39 AM, S G <
> > > > >>>> sg.online.email@gmail.com <ma...@gmail.com>
> > > > >>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks Bobby,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> We will need some kind of IMetricsConsumer to talk to
> > > telegraf.
> > > > >>>>>>>>>> Many other softwares like Solr, Elastic-Search, Cassandra
> > etc.
> > > > >>>> provide
> > > > >>>>>>>>>> metrics through a URL making it very easy to consume by
> > tools
> > > > >> like
> > > > >>>>>>>> telegraf.
> > > > >>>>>>>>>> How about a IMetricsConsumer that will run on storm-ui and
> > > > >> provide
> > > > >>>> the
> > > > >>>>>>>>>> metrics through a URL such as <storm-ui-host>/metrics ?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Also, I see the following option in defaults.yaml:
> > > > >>>>>>>>>> #default storm daemon metrics reporter plugins
> > > > >>>>>>>>>> storm.daemon.metrics.reporter.plugins:
> > > > >>>>>>>>>> - "org.apache.storm.daemon.metrics.reporters.
> > > > >>>>>>>> JmxPreparableReporter"
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Is this a good option to use for converting metrics into
> > JMX ?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks
> > > > >>>>>>>>>> SG
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans
> > > > >>>>>> <evans@yahoo-inc.com.invalid <mailto:evans@yahoo-inc.com.
> > invalid>
> > > > >>>>>>>>>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> HttpForwardingMetricsServer is a real hack intended really
> > for
> > > > >>>>>> tests. I
> > > > >>>>>>>>>> know I wrote it :). Please don't use it in production.
> You
> > > can
> > > > >>>> write
> > > > >>>>>>>> your
> > > > >>>>>>>>>> own IMetricesConsumer to do whatever you want to with the
> > > > >> metrics.
> > > > >>>>>>>>>> https://github.com/apache/ <https://github.com/apache/>
> > > storm/blob/master/storm-core/
> > > > >>>>>>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer.
> java
> > > > >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/
> <
> > > https://github.com/apache/storm/blob/master/storm-core/>
> > > > >>>>>>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> That is the correct way to get the data out. If you want
> to
> > > > >>> write a
> > > > >>>>>>>>>> bridge to JMX for this that might work, but going directly
> > to
> > > > >>>>>> telegraph
> > > > >>>>>>>>>> would probably be better. - Bobby
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Thursday, October 6, 2016 1:43 PM, S G <
> > > > >>>>>>>> sg.online.email@gmail.com <mailto:sg.online.email@gmail.com
> >>
> > > > >>>>>>>>>> wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Hi,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> We want to use Telegraf (
> > > > >>>>>>>>>> https://github.com/influxdata/ <https://github.com/
> > > influxdata/>telegraf/tree/master/plugins
> > > > >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/
> master/plugins
> > <
> > > https://github.com/influxdata/telegraf/tree/master/plugins>>)
> > > > >> for
> > > > >>>>>>>> getting
> > > > >>>>>>>>>> storm's metrics.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> But we do not want to add a HttpForwardingMetricsServer
> just
> > > to
> > > > >>> get
> > > > >>>>>> the
> > > > >>>>>>>>>> metrics and send them to telegraf.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Other option is to use Jolokia (https://jolokia.org/ <
> > > https://jolokia.org/>) that can
> > > > >>> read
> > > > >>>>>> JMX
> > > > >>>>>>>>>> and
> > > > >>>>>>>>>> write into telegraf.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Does storm report all its metrics (including those of
> custom
> > > > >>>>>>>> spouts/bolts)
> > > > >>>>>>>>>> into JMX?
> > > > >>>>>>>>>> Or spawning a HttpForwardingMetricsServer is the only
> > option?
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Thanks
> > > > >>>>>>>>>> SG
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>>
> > > > >>>>>>>
> > > > >>>>>>
> > > > >>>>
> > > > >>>>
> > > > >>>>
> > > > >>>
> > > > >>>
> > > > >>>
> > >
> >
>
Re: Are storm metrics reported through JMX too?
Posted by Bobby Evans <ev...@yahoo-inc.com.INVALID>.
For me the big issue is that not every process has a web server on it. That may change in the future but for now only the ui and the logviewer have a web server up and running. What is more if we wanted to do this for the workers we would need to think about an alternative port that would be free for a web server to be on, the resources a web server would use inside the worker and how would authentication work to those worker processes. SPNEGO would be really difficult to make work.
- Bobby
On Tuesday, February 7, 2017, 9:57:27 PM CST, Tech Id <te...@gmail.com> wrote:Hi Alessandro, Taylor,
Any more updates on this one?
This seems like a very good feature to have.
Thanks
TI
On Tue, Dec 6, 2016 at 10:37 AM, Alessandro Bellina <
abellina@yahoo-inc.com.invalid> wrote:
> Hi S G,
> Not something I am working on now. What I pushed was just reporter config
> for Taylor since he needed it. I do think that if default reporting works
> we could just go to the rocksdb store and ask it for the metrics there?
> There are other parts of this I haven't published yet but looking to do so
> either this or next week.
> Thanks,
> Alessandro
>
>
>
> On Tuesday, December 6, 2016, 11:10:57 AM CST, S G <
> sg.online.email@gmail.com> wrote:Hey Alessandro,
>
> Thanks for sharing.
> Please share if we have plans to use the metrics-servlets from dropwizard?
> (http://metrics.dropwizard.io/3.1.0/manual/servlets/#adminservlet)
>
> I think it will be very convenient to have the metrics reported through a
> REST API from every process (worker, supervisor, nimbus etc.) and in line
> with most other software like Solr, ES etc. This can be achieved very
> easily if we use the above metric servlets and they also provide ping,
> health-check, thread dump etc which are useful too.
>
> Please ignore if its already there in the code shared above and I could not
> find it.
>
> Thanks,
> SG
>
>
>
> On Mon, Dec 5, 2016 at 9:49 PM, Alessandro Bellina <abellina@yahoo-inc.com
> >
> wrote:
>
> > Hi Taylor
> >
> > Please see latest commit in: https://github.com/abellina/
> > storm/tree/reporters
> >
> > Specifically inside: storm-core/src/jvm/org/apache/storm/metrics2
> >
> > I have a default config that sets up a couple of reporters in
> > default.yaml. The format is inline with what we discussed, but updated
> with
> > what I suggested over the weekend.
> >
> > This is definitely a work in progress, but you should be able to
> > instantiate reporters for your purposes.
> >
> > Thanks,
> >
> > Alessandro
> >
> >
> >
> > On Monday, December 5, 2016, 1:26:41 PM CST, Alessandro Bellina <
> > abellina@yahoo-inc.com.INVALID> wrote:
> > Yes. Will PR this tonight Taylor. Thanks!
> > On Monday, December 5, 2016, 12:37:18 PM CST, P. Taylor Goetz <
> > ptgoetz@gmail.com> wrote:Alessandro,
> >
> > Are you in a position to open a pull request against the metrics_v2
> > branch? I’d like to start integrating the work I’ve been doing with the
> > reporter configuration stuff you have. If what you have is
> incomplete/WIP,
> > that’s not a big deal as the metrics_v2 branch is a feature branch and
> > we’ll have plenty of opportunities to clean things up.
> >
> > -Taylor
> >
> > > On Nov 29, 2016, at 3:27 PM, Alessandro Bellina <
> abellina@yahoo-inc.com>
> > wrote:
> > >
> > > Taylor,
> > >
> > > Ok maybe there is some effort duplication. For the config, I have the
> > bare minimum to get the default reporter up. I'll focus on that since you
> > could use it. Will update JIRA with more.
> > >
> > > Alessandro
> > >
> > >
> > > ----- Forwarded Message -----
> > > From: P. Taylor Goetz <pt...@gmail.com>
> > > To: "dev@storm.apache.org" <de...@storm.apache.org>
> > > Cc: S G <sg...@gmail.com>; naren@narendasan.com <
> > naren@narendasan.com>; Austin Chung <ac...@illinois.edu>
> > > Sent: Tuesday, November 29, 2016, 1:27:58 PM CST
> > > Subject: Re: Are storm metrics reported through JMX too?
> > >
> > > Alessandro,
> > >
> > > Where do you stand with the reporter configuration via the storm.yaml
> > config file?
> > >
> > > I have metrics collection for workers and disruptor queues almost
> ready,
> > but now I’m looking for flexible configuration (right now I have
> reporters
> > hard coded).
> > >
> > > -Taylor
> > >
> > > > On Nov 29, 2016, at 1:57 PM, Alessandro Bellina <
> > abellina@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>>
> > wrote:
> > > >
> > > > S G,
> > > > I am slowly making progress and plan to share something this week,
> > specifically for hooking up a default codahale reporter to the workers.
> > Also the U of I students for the open source class have made progress on
> > their rocksdb implementation for the default store (the stuff I am doing
> > should feed into their store). I don't think any of this is going to be
> in
> > 1.1.0 given that it hasn't really been looked at by others, let alone
> > reviewed.
> > > > Thanks
> > > > Alessandro
> > > > On Monday, November 28, 2016, 4:51:34 PM CST, S G <
> > sg.online.email@gmail.com <ma...@gmail.com>> wrote:Hi,
> > > >
> > > > I see https://issues.apache.org/jira/browse/STORM-2153 <
> > https://issues.apache.org/jira/browse/STORM-2153>being open for this
> > > > and a lot of good discussion happening too.
> > > > There is also a talk for releasing 1.1.0 version soon.
> > > >
> > > > So I just wanted to check if any metric related improvements will be
> > > > available in 1.1.0.
> > > > It would be great to try out these new improvements.
> > > >
> > > > Thanks
> > > > SG
> > > >
> > > > On Tue, Oct 18, 2016 at 8:57 PM, Abhishek Nigam <adn5327@gmail.com
> > <ma...@gmail.com>> wrote:
> > > >
> > > >> Hi all,
> > > >>
> > > >> I'm Abhishek Nigam, one of the students from the group at the
> > University of
> > > >> Illinois; I'm really looking forward to working on this issue! We'll
> > be
> > > >> sure to keep you all updated as to how we progress.
> > > >>
> > > >> Cheers,
> > > >> --
> > > >> Abhishek
> > > >>
> > > >> On Tue, Oct 18, 2016 at 1:23 PM Alessandro Bellina <
> > abellina@yahoo-inc.com <ma...@yahoo-inc.com>
> > > >>>
> > > >> wrote:
> > > >>
> > > >>> + Naren, Austin, and Abhishek, the students from the University of
> > > >>> Illinois Open Source class.
> > > >>>
> > > >>>
> > > >>> On Tuesday, October 11, 2016 11:48 PM, S G <
> > sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>> wrote:
> > > >>>
> > > >>>
> > > >>> Response on this important issue is pretty good. I am happily
> > surprised
> > > >> :)
> > > >>>
> > > >>> I want to mention our strategy for extracting metrics from other
> > > >> products.
> > > >>> We use jolokia_proxy (https://jolokia.org/features/proxy.html <
> > https://jolokia.org/features/proxy.html>) to get
> > > >> JMX
> > > >>> beans from several softwares and feed them to telegraf. That way,
> we
> > > >> avoid
> > > >>> writing custom processors for all these different products.
> > > >>>
> > > >>> Telegraf is quickly becoming a standard for metrics data. (Just see
> > the
> > > >>> list of input plugins here:
> > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/inputs
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins/inputs>). And
> > > >> it
> > > >>> integrates well with several outputs too (
> > > >>> https://github.com/influxdata/telegraf/tree/master/plugins/outputs
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins/outputs>).
> > > >>>
> > > >>> Also since the metrics in JMX, they can be queried by jolokia-agent
> > > >>> installed per node. This avoids the extra metrics-consumer bolt
> > which can
> > > >>> hit the topology throughtput too.
> > > >>>
> > > >>> So I cast my vote in favor of JMX-implementation of metrics.
> > > >>> Other approaches are welcome to be discussed.
> > > >>>
> > > >>> On Tue, Oct 11, 2016 at 7:30 PM, Alessandro Bellina <
> > > >>> abellina@yahoo-inc.com.invalid <mailto:abellina@yahoo-inc.
> > com.invalid>> wrote:
> > > >>>
> > > >>>> blockquote, div.yahoo_quoted { margin-left: 0 !important;
> > > >>> border-left:1px
> > > >>>> #715FFA solid !important; padding-left:1ex !important;
> > > >>>> background-color:white !important; } Yeap that's a requirement
> from
> > our
> > > >>>> perspective (working through this list).
> > > >>>> Sure I think as usual we can start with master with an eye for
> what
> > > >> would
> > > >>>> need to be back ported.
> > > >>>> Sent from Yahoo Mail for iPhone
> > > >>>>
> > > >>>>
> > > >>>> On Tuesday, October 11, 2016, 8:50 PM, P. Taylor Goetz <
> > > >>> ptgoetz@gmail.com <ma...@gmail.com>>
> > > >>>> wrote:
> > > >>>>
> > > >>>> I hope I didn't come across as overly critical. You did the best
> > with
> > > >>> what
> > > >>>> you had to work with. Which isn't pretty.
> > > >>>>
> > > >>>> We could potentially do a parallel metrics API in 1.1, 1.2, or
> > master
> > > >> and
> > > >>>> still stay close to semantic versioning...?
> > > >>>>
> > > >>>> -Taylor
> > > >>>>
> > > >>>>> On Oct 11, 2016, at 9:28 PM, Jungtaek Lim <kabhwan@gmail.com
> > <ma...@gmail.com>> wrote:
> > > >>>>>
> > > >>>>> Yeah I admit that configuration flag was bad for me also, but I
> > have
> > > >> no
> > > >>>>> alternatives. Only way to avoid struggling with design limitation
> > is
> > > >>>> revamp
> > > >>>>> / redesign.
> > > >>>>> Thanks S G for exposing willingness of volunteer and great news
> > > >>>> Alessandro
> > > >>>>> for that project.
> > > >>>>> Alessandro, could you forward the upcoming news for the project
> to
> > > >> dev@
> > > >>>>> list?
> > > >>>>>
> > > >>>>> - Jungtaek Lim (HeartSaVioR)
> > > >>>>>
> > > >>>>> 2016년 10월 12일 (수) 오전 10:22, P. Taylor Goetz <ptgoetz@gmail.com
> > <ma...@gmail.com>>님이
> > > >> 작성:
> > > >>>>>
> > > >>>>>> I was thinking on a smaller scale in terms of effort, but the
> > more I
> > > >>>> think
> > > >>>>>> about it, the more supportive I would be of a full revamp (new
> > API)
> > > >>> for
> > > >>>>>> metrics based on Coda Hale's metrics library. It's proven and
> > > >> stable.
> > > >>>> I've
> > > >>>>>> used it many times. I think either approach would be roughly the
> > > >> same
> > > >>>>>> amount of work.
> > > >>>>>>
> > > >>>>>> Some of the metrics API improvements in the 1.1.x branch are
> nice,
> > > >> but
> > > >>>>>> IMHO are lipstick on a pig.
> > > >>>>>>
> > > >>>>>> With apologies to Jungtaek, who has done amazing work all across
> > the
> > > >>>>>> codebase, I'm a little squeamish about the proposed change to
> > > >> metrics
> > > >>>> that
> > > >>>>>> changes the consumer API based on a configuration flag (don't
> know
> > > >> the
> > > >>>> PR
> > > >>>>>> number offhand).
> > > >>>>>>
> > > >>>>>> I'm +1 for moving in this direction (revamped metrics). Let's
> end
> > > >> the
> > > >>>>>> metrics pain.
> > > >>>>>>
> > > >>>>>> -Taylor
> > > >>>>>>
> > > >>>>>>> On Oct 11, 2016, at 10:15 AM, Bobby Evans <
> > > >>> evans@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>
> > > >>>>>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>> I agree that IMetricsConsumer is not good, but the reality is
> > that
> > > >>> all
> > > >>>>>> of the metrics system needs to be redone. The problem is that
> we
> > > >> ship
> > > >>>> an
> > > >>>>>> object as a metric. If I get an object I have no idea what it
> is
> > > >> hand
> > > >>>>>> hence no idea how to report it or what to do with it. What is
> > more
> > > >>> the
> > > >>>>>> common types we use in the metrics we provide are really not
> > enough.
> > > >>>> For
> > > >>>>>> example CountMetric sends a Long. Well when I get it in the
> > metrics
> > > >>>>>> consumer I have no idea if I should report it like a counter or
> > if I
> > > >>>> should
> > > >>>>>> report it like a gauge (something that every metrics system I
> have
> > > >>> used
> > > >>>>>> wants to know). But then we support pre-aggregation of the
> > metrics
> > > >>> with
> > > >>>>>> IReducer so the number I get might be an average instead of
> > either a
> > > >>>> gauge
> > > >>>>>> or a counter, which no good metrics system will want to collect
> > > >>> because
> > > >>>> I
> > > >>>>>> cannot aggregate it with anything else, the math just does not
> > work.
> > > >>>>>>> The proposal I have said before and I still believe is that we
> > need
> > > >>> to
> > > >>>>>> put in place a parallel metrics API/system. We will deprecate
> all
> > > >> of
> > > >>>>>> https://git.corp.yahoo.com/storm/storm/tree/master- <
> > https://git.corp.yahoo.com/storm/storm/tree/master->
> > > >>>> security/storm-core/src/jvm/backtype/storm/metric/api
> > > >>>>>> and create a new parallel one that provides an API similar to
> > > >>>>>> http://metrics.dropwizard.io/3.1.0/.
> > > >>> <http://metrics.dropwizard.io/3.1.0/ <
> http://metrics.dropwizard.io/
> > 3.1.0/>> I would even be fine in just
> > > >>>> using
> > > >>>>>> their API and exposing that to end users. Dropwizard has solved
> > all
> > > >>> of
> > > >>>>>> these problems already and I don't see a reason to reinvent the
> > > >> wheel.
> > > >>>> I
> > > >>>>>> don't personally see a lot of value in trying to send all of the
> > > >>> metrics
> > > >>>>>> through storm itslef. I am fine if we are able to support that,
> > but
> > > >>> it
> > > >>>> is
> > > >>>>>> far from a requirement. - Bobby
> > > >>>>>>>
> > > >>>>>>> On Monday, October 10, 2016 10:47 PM, S G <
> > > >>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>> wrote:
> > > >>>>>>>
> > > >>>>>>>
> > > >>>>>>> +1
> > > >>>>>>> We can probably start by opening a JIRA for this and adding a
> > > >> design
> > > >>>>>>> approach for the same?
> > > >>>>>>> I would like to help in the coding-effort for this.
> > > >>>>>>>
> > > >>>>>>> -SG
> > > >>>>>>>
> > > >>>>>>>> On Mon, Oct 10, 2016 at 1:51 PM, P. Taylor Goetz <
> > > >> ptgoetz@gmail.com <ma...@gmail.com>
> > > >>>>
> > > >>>>>> wrote:
> > > >>>>>>>>
> > > >>>>>>>> I’ve been thinking about metrics lately, specifically the fact
> > > >> that
> > > >>>>>> people
> > > >>>>>>>> tend to struggle with implementing a metrics consumer. (Like
> > this
> > > >>> one
> > > >>>>>> [1]).
> > > >>>>>>>>
> > > >>>>>>>> The IMetricsConsumer interface is pretty low level, and common
> > > >>>>>>>> aggregations, calculations, etc. are left up to each
> individual
> > > >>>>>>>> implementation. That seems like an area where further
> > abstraction
> > > >>>> would
> > > >>>>>>>> make it easier to support different back ends (Graphite, JMX,
> > > >>> Splunk,
> > > >>>>>> etc.).
> > > >>>>>>>>
> > > >>>>>>>> My thought is to create an abstract IMetricsConsumer
> > > >> implementation
> > > >>>> that
> > > >>>>>>>> does common aggregations and calculations, and then delegates
> > to a
> > > >>>>>> plugable
> > > >>>>>>>> “metrics sink” implementation (e.g. “IMetricsSink”, etc.).
> That
> > > >>> would
> > > >>>>>>>> greatly simplify the effort required to integrate with various
> > > >>>> external
> > > >>>>>>>> metrics systems. I know of at least a few users that would be
> > > >>>>>> interested,
> > > >>>>>>>> one is currently scraping the logs from LoggingMetricsConsumer
> > and
> > > >>>>>> polling
> > > >>>>>>>> the Storm REST API for their metrics.
> > > >>>>>>>>
> > > >>>>>>>> -Taylor
> > > >>>>>>>>
> > > >>>>>>>> [1] http://twocentsonsoftware.blogspot.co.il/2014/12/ <
> > http://twocentsonsoftware.blogspot.co.il/2014/12/>
> > > >>>>>>>> sending-out-storm-metrics.html
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>> On Oct 10, 2016, at 12:14 PM, Bobby Evans
> > > >>>> <evans@yahoo-inc.com.INVALID <ma...@yahoo-inc.com.INVALID>
> > > >>>>>>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>> First of all the server exposes essentially the same
> interface
> > > >> that
> > > >>>> the
> > > >>>>>>>> IMetricsConsumer exposes. It mostly just adds a bunch of
> > overhead
> > > >>> in
> > > >>>>>> the
> > > >>>>>>>> middle to serialize out the objects send them over http to
> > another
> > > >>>>>> process
> > > >>>>>>>> which then has to deserialize them and process them. If you
> > > >> really
> > > >>>>>> don't
> > > >>>>>>>> need the metrics to show up on a special known box you can
> have
> > > >> that
> > > >>>>>> exact
> > > >>>>>>>> same code running inside the metrics consumer without all of
> the
> > > >>>>>> overhead.
> > > >>>>>>>>> The server/client are insecure, have to deal with thread
> issues
> > > >>> that
> > > >>>> a
> > > >>>>>>>> normal IMetricsConsumer does not, and are not written to be
> > robust
> > > >>> (If
> > > >>>>>> the
> > > >>>>>>>> HTTP server is down the consumer crashes and continues to
> crash
> > > >>> until
> > > >>>>>> the
> > > >>>>>>>> server is brought back up). It was written very quickly for a
> > > >> test
> > > >>>>>>>> situation and it honestly never crossed my mind that anyone
> > would
> > > >>> want
> > > >>>>>> to
> > > >>>>>>>> use it in production.
> > > >>>>>>>>>
> > > >>>>>>>>> - Bobby
> > > >>>>>>>>>
> > > >>>>>>>>> On Monday, October 10, 2016 10:59 AM, S G <
> > > >>>>>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>> Thanks Bobby.
> > > >>>>>>>>>
> > > >>>>>>>>> If we write our own metrics consumer, how do we ensure that
> it
> > is
> > > >>>>>> better
> > > >>>>>>>>> than HttpForwardingMetricsServer? In other words, what
> aspects
> > of
> > > >>> the
> > > >>>>>>>>> HttpForwardingMetricsServer
> > > >>>>>>>>> should we avoid to make our own metrics consumer better and
> > ready
> > > >>> for
> > > >>>>>>>>> production?
> > > >>>>>>>>>
> > > >>>>>>>>> Is versign/storm-graphite <https://github.com/verisign/ <
> > https://github.com/verisign/>
> > > >>>> storm-graphite>
> > > >>>>>>>>> production
> > > >>>>>>>>> ready?
> > > >>>>>>>>>
> > > >>>>>>>>> Also, we should add a line about production-readiness of
> > > >>>>>>>>> HttpForwardingMetricsServer
> > > >>>>>>>>> in the documentation at http://storm.apache.org/ <
> > http://storm.apache.org/>
> > > >>>>>>>> releases/1.0.2/Metrics.html
> > > >>>>>>>>> (We were just about to think seriously on using this for
> > > >> production
> > > >>>> as
> > > >>>>>> we
> > > >>>>>>>>> thought this to be the standard solution for metrics'
> > consumption
> > > >>> in
> > > >>>>>> 1.0+
> > > >>>>>>>>> version).
> > > >>>>>>>>>
> > > >>>>>>>>> -SG
> > > >>>>>>>>>
> > > >>>>>>>>> On Mon, Oct 10, 2016 at 6:37 AM, Bobby Evans <
> > > >> evans@yahoo-inc.com <ma...@yahoo-inc.com>>
> > > >>>>>>>> wrote:
> > > >>>>>>>>>
> > > >>>>>>>>>> First of all there really are two different sets of metrics.
> > > >> One
> > > >>>> set
> > > >>>>>> is
> > > >>>>>>>>>> the topology metrics and the other set is the daemon metrics
> > > >>>> (metrics
> > > >>>>>>>> for
> > > >>>>>>>>>> things like the ui and nimbus). The JmxPreparableReporter
> > > >> plugin
> > > >>>> only
> > > >>>>>>>>>> exposes daemon metrics not the topology metrics through JMX.
> > > >>>> Exposing
> > > >>>>>>>>>> topology metrics through JMX is a non trivial task. The
> > current
> > > >>>>>> metrics
> > > >>>>>>>>>> feature was not designed for this. We are in the process of
> > > >>> trying
> > > >>>> to
> > > >>>>>>>>>> redesign the metrics system to allow for features like this,
> > but
> > > >>> it
> > > >>>> is
> > > >>>>>>>>>> still a ways off.
> > > >>>>>>>>>>
> > > >>>>>>>>>> - Bobby
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Saturday, October 8, 2016 11:39 AM, S G <
> > > >>>> sg.online.email@gmail.com <ma...@gmail.com>
> > > >>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks Bobby,
> > > >>>>>>>>>>
> > > >>>>>>>>>> We will need some kind of IMetricsConsumer to talk to
> > telegraf.
> > > >>>>>>>>>> Many other softwares like Solr, Elastic-Search, Cassandra
> etc.
> > > >>>> provide
> > > >>>>>>>>>> metrics through a URL making it very easy to consume by
> tools
> > > >> like
> > > >>>>>>>> telegraf.
> > > >>>>>>>>>> How about a IMetricsConsumer that will run on storm-ui and
> > > >> provide
> > > >>>> the
> > > >>>>>>>>>> metrics through a URL such as <storm-ui-host>/metrics ?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Also, I see the following option in defaults.yaml:
> > > >>>>>>>>>> #default storm daemon metrics reporter plugins
> > > >>>>>>>>>> storm.daemon.metrics.reporter.plugins:
> > > >>>>>>>>>> - "org.apache.storm.daemon.metrics.reporters.
> > > >>>>>>>> JmxPreparableReporter"
> > > >>>>>>>>>>
> > > >>>>>>>>>> Is this a good option to use for converting metrics into
> JMX ?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks
> > > >>>>>>>>>> SG
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Fri, Oct 7, 2016 at 8:11 AM, Bobby Evans
> > > >>>>>> <evans@yahoo-inc.com.invalid <mailto:evans@yahoo-inc.com.
> invalid>
> > > >>>>>>>>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>> HttpForwardingMetricsServer is a real hack intended really
> for
> > > >>>>>> tests. I
> > > >>>>>>>>>> know I wrote it :). Please don't use it in production. You
> > can
> > > >>>> write
> > > >>>>>>>> your
> > > >>>>>>>>>> own IMetricesConsumer to do whatever you want to with the
> > > >> metrics.
> > > >>>>>>>>>> https://github.com/apache/ <https://github.com/apache/>
> > storm/blob/master/storm-core/
> > > >>>>>>>>>> src/jvm/org/apache/storm/ metric/api/IMetricsConsumer. java
> > > >>>>>>>>>> <https://github.com/apache/storm/blob/master/storm-core/ <
> > https://github.com/apache/storm/blob/master/storm-core/>
> > > >>>>>>>> src/jvm/org/apache/storm/metric/api/IMetricsConsumer.java>
> > > >>>>>>>>>>
> > > >>>>>>>>>> That is the correct way to get the data out. If you want to
> > > >>> write a
> > > >>>>>>>>>> bridge to JMX for this that might work, but going directly
> to
> > > >>>>>> telegraph
> > > >>>>>>>>>> would probably be better. - Bobby
> > > >>>>>>>>>>
> > > >>>>>>>>>> On Thursday, October 6, 2016 1:43 PM, S G <
> > > >>>>>>>> sg.online.email@gmail.com <ma...@gmail.com>>
> > > >>>>>>>>>> wrote:
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>> Hi,
> > > >>>>>>>>>>
> > > >>>>>>>>>> We want to use Telegraf (
> > > >>>>>>>>>> https://github.com/influxdata/ <https://github.com/
> > influxdata/>telegraf/tree/master/plugins
> > > >>>>>>>>>> <https://github.com/influxdata/telegraf/tree/master/plugins
> <
> > https://github.com/influxdata/telegraf/tree/master/plugins>>)
> > > >> for
> > > >>>>>>>> getting
> > > >>>>>>>>>> storm's metrics.
> > > >>>>>>>>>>
> > > >>>>>>>>>> But we do not want to add a HttpForwardingMetricsServer just
> > to
> > > >>> get
> > > >>>>>> the
> > > >>>>>>>>>> metrics and send them to telegraf.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Other option is to use Jolokia (https://jolokia.org/ <
> > https://jolokia.org/>) that can
> > > >>> read
> > > >>>>>> JMX
> > > >>>>>>>>>> and
> > > >>>>>>>>>> write into telegraf.
> > > >>>>>>>>>>
> > > >>>>>>>>>> Does storm report all its metrics (including those of custom
> > > >>>>>>>> spouts/bolts)
> > > >>>>>>>>>> into JMX?
> > > >>>>>>>>>> Or spawning a HttpForwardingMetricsServer is the only
> option?
> > > >>>>>>>>>>
> > > >>>>>>>>>> Thanks
> > > >>>>>>>>>> SG
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>>
> > > >>>>>>>
> > > >>>>>>
> > > >>>>
> > > >>>>
> > > >>>>
> > > >>>
> > > >>>
> > > >>>
> >
>