You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by liu brent <li...@gmail.com> on 2012/04/09 09:30:08 UTC
Use Kafka to collect JVM and JMX MBean's statistics.
Hi,
Currently, there are 20+ kinds of web service running on hundreds of
machines in 4 data centers, which are written in java. We want to collect
and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
and machines, and then send the waring to Nagios. Does anyone has the
experience in using Kafka to do so?
The direct solution seems like that we install a client on every machine,
which works as JMX client, collect the information and work as kafka
producer. Anyone has better ideas or suggestions.
Thanks,
Liu
Re: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by Jun Rao <ju...@gmail.com>.
Liu,
We use Kafka to collect Mbean stats at LinkedIn. Basically, we have some
wrapper code that periodically reads Mbean values and writes them to Kafka
using a producer.
Thanks,
Jun
On Mon, Apr 9, 2012 at 12:30 AM, liu brent <li...@gmail.com> wrote:
> Hi,
> Currently, there are 20+ kinds of web service running on hundreds of
> machines in 4 data centers, which are written in java. We want to collect
> and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> and machines, and then send the waring to Nagios. Does anyone has the
> experience in using Kafka to do so?
> The direct solution seems like that we install a client on every machine,
> which works as JMX client, collect the information and work as kafka
> producer. Anyone has better ideas or suggestions.
>
> Thanks,
> Liu
>
Re: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by Pierre-Yves Ritschard <py...@spootnik.org>.
I would also encourage looking into collectd which has a great JMX
input plugin, the java bindings for collectd also allow very easy
integration of kafka for the output
(an interface with a single method must be implemented, which gets a
list of values).
See:
* http://collectd.org/wiki/index.php/Plugin:GenericJMX (how to query JMX values)
* http://collectd.org/wiki/index.php/Plugin:Java (how to register a
kafka plugin)
* http://collectd.org/documentation/javadoc/ (java interfaces to implement)
* http://collectd.org (project home page, much more than jmx can be
fed to kafka)
On Mon, Apr 16, 2012 at 5:16 AM, liu brent <li...@gmail.com> wrote:
> Thanks guys, I think tcollector meets our needs now. :)
>
> On Tue, Apr 10, 2012 at 12:43 AM, Dave Barr <da...@gmail.com> wrote:
>
>> We have an agent (tcollector) that collects various metrics and sends
>> them to OpenTSDB. We then have a simple python script that lets you
>> query OpenTSDB for a given metric and alert on a threshold. We
>> currently gather about 1300 metrics and about 10000 datapoints/sec.
>>
>> https://github.com/stumbleupon/tcollector
>> http://opentsdb.net/nagios.html
>>
>> We're just starting to use Kafka here. I'm not aware yet of a Kafka
>> collector, but I'm sure one will be coming soon. They're really
>> simple to write. You can look at either the elasticsearch collector
>> or the HBase collector as an example of different styles. The HBase
>> one forks a jmx process to read metrics from the running JVM, the
>> elasticsearch one queries the running process for metrics via an
>> ElasticSearch endpoint.
>>
>> --Dave
>> On Mon, Apr 9, 2012 at 12:30 AM, liu brent <li...@gmail.com> wrote:
>> > Hi,
>> > Currently, there are 20+ kinds of web service running on hundreds of
>> > machines in 4 data centers, which are written in java. We want to collect
>> > and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
>> > and machines, and then send the waring to Nagios. Does anyone has the
>> > experience in using Kafka to do so?
>> > The direct solution seems like that we install a client on every
>> machine,
>> > which works as JMX client, collect the information and work as kafka
>> > producer. Anyone has better ideas or suggestions.
>> >
>> > Thanks,
>> > Liu
>>
Re: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by liu brent <li...@gmail.com>.
Thanks guys, I think tcollector meets our needs now. :)
On Tue, Apr 10, 2012 at 12:43 AM, Dave Barr <da...@gmail.com> wrote:
> We have an agent (tcollector) that collects various metrics and sends
> them to OpenTSDB. We then have a simple python script that lets you
> query OpenTSDB for a given metric and alert on a threshold. We
> currently gather about 1300 metrics and about 10000 datapoints/sec.
>
> https://github.com/stumbleupon/tcollector
> http://opentsdb.net/nagios.html
>
> We're just starting to use Kafka here. I'm not aware yet of a Kafka
> collector, but I'm sure one will be coming soon. They're really
> simple to write. You can look at either the elasticsearch collector
> or the HBase collector as an example of different styles. The HBase
> one forks a jmx process to read metrics from the running JVM, the
> elasticsearch one queries the running process for metrics via an
> ElasticSearch endpoint.
>
> --Dave
> On Mon, Apr 9, 2012 at 12:30 AM, liu brent <li...@gmail.com> wrote:
> > Hi,
> > Currently, there are 20+ kinds of web service running on hundreds of
> > machines in 4 data centers, which are written in java. We want to collect
> > and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> > and machines, and then send the waring to Nagios. Does anyone has the
> > experience in using Kafka to do so?
> > The direct solution seems like that we install a client on every
> machine,
> > which works as JMX client, collect the information and work as kafka
> > producer. Anyone has better ideas or suggestions.
> >
> > Thanks,
> > Liu
>
Re: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by Dave Barr <da...@gmail.com>.
We have an agent (tcollector) that collects various metrics and sends
them to OpenTSDB. We then have a simple python script that lets you
query OpenTSDB for a given metric and alert on a threshold. We
currently gather about 1300 metrics and about 10000 datapoints/sec.
https://github.com/stumbleupon/tcollector
http://opentsdb.net/nagios.html
We're just starting to use Kafka here. I'm not aware yet of a Kafka
collector, but I'm sure one will be coming soon. They're really
simple to write. You can look at either the elasticsearch collector
or the HBase collector as an example of different styles. The HBase
one forks a jmx process to read metrics from the running JVM, the
elasticsearch one queries the running process for metrics via an
ElasticSearch endpoint.
--Dave
On Mon, Apr 9, 2012 at 12:30 AM, liu brent <li...@gmail.com> wrote:
> Hi,
> Currently, there are 20+ kinds of web service running on hundreds of
> machines in 4 data centers, which are written in java. We want to collect
> and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> and machines, and then send the waring to Nagios. Does anyone has the
> experience in using Kafka to do so?
> The direct solution seems like that we install a client on every machine,
> which works as JMX client, collect the information and work as kafka
> producer. Anyone has better ideas or suggestions.
>
> Thanks,
> Liu
RE: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by Mathias Herberts <ma...@gmail.com>.
it's not, read the slides flume 0.9.x used a push model, Kagka enables
decoupling consumers and producers, pretty much what flume ng ended up
dokng with channels.
We've been collecting 2B+ metrics per day for almost a year without
problems.
On Apr 9, 2012 5:42 PM, "VERMEERBERGEN Alexandre" <
Alexandre.VERMEERBERGEN@3ds.com> wrote:
> Surprenant: un usage combiné de Flume (devenu comme Kafka un incubateur
> Apache) et de Kafka.
> Cela me semble un peu redondant...
>
>
> -----Original Message-----
> From: Mathias Herberts [mailto:mathias.herberts@gmail.com]
> Sent: Monday, April 09, 2012 4:52 PM
> To: kafka-users@incubator.apache.org
> Subject: Re: Use Kafka to collect JVM and JMX MBean's statistics.
>
> At Crédit Mutuel Arkéa we collect metrics among which JMX ones using Flume
> and a custom Kafka Sink.
>
> Search for 'Artimon' on slideshare.
> On Apr 9, 2012 9:30 AM, "liu brent" <li...@gmail.com> wrote:
>
> > Hi,
> > Currently, there are 20+ kinds of web service running on hundreds of
> > machines in 4 data centers, which are written in java. We want to collect
> > and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> > and machines, and then send the waring to Nagios. Does anyone has the
> > experience in using Kafka to do so?
> > The direct solution seems like that we install a client on every
> machine,
> > which works as JMX client, collect the information and work as kafka
> > producer. Anyone has better ideas or suggestions.
> >
> > Thanks,
> > Liu
> >
> This email and any attachments are intended solely for the use of the
> individual or entity to whom it is addressed and may be confidential and/or
> privileged.
>
> If you are not one of the named recipients or have received this email in
> error,
>
> (i) you should not read, disclose, or copy it,
>
> (ii) please notify sender of your receipt by reply email and delete this
> email and all attachments,
>
> (iii) Dassault Systemes does not accept or assume any liability or
> responsibility for any use of or reliance on this email.
>
> For other languages, go to http://www.3ds.com/terms/email-disclaimer
>
RE: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by VERMEERBERGEN Alexandre <Al...@3ds.com>.
Surprenant: un usage combiné de Flume (devenu comme Kafka un incubateur Apache) et de Kafka.
Cela me semble un peu redondant...
-----Original Message-----
From: Mathias Herberts [mailto:mathias.herberts@gmail.com]
Sent: Monday, April 09, 2012 4:52 PM
To: kafka-users@incubator.apache.org
Subject: Re: Use Kafka to collect JVM and JMX MBean's statistics.
At Crédit Mutuel Arkéa we collect metrics among which JMX ones using Flume
and a custom Kafka Sink.
Search for 'Artimon' on slideshare.
On Apr 9, 2012 9:30 AM, "liu brent" <li...@gmail.com> wrote:
> Hi,
> Currently, there are 20+ kinds of web service running on hundreds of
> machines in 4 data centers, which are written in java. We want to collect
> and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> and machines, and then send the waring to Nagios. Does anyone has the
> experience in using Kafka to do so?
> The direct solution seems like that we install a client on every machine,
> which works as JMX client, collect the information and work as kafka
> producer. Anyone has better ideas or suggestions.
>
> Thanks,
> Liu
>
This email and any attachments are intended solely for the use of the individual or entity to whom it is addressed and may be confidential and/or privileged.
If you are not one of the named recipients or have received this email in error,
(i) you should not read, disclose, or copy it,
(ii) please notify sender of your receipt by reply email and delete this email and all attachments,
(iii) Dassault Systemes does not accept or assume any liability or responsibility for any use of or reliance on this email.
For other languages, go to http://www.3ds.com/terms/email-disclaimer
Re: Use Kafka to collect JVM and JMX MBean's statistics.
Posted by Mathias Herberts <ma...@gmail.com>.
At Crédit Mutuel Arkéa we collect metrics among which JMX ones using Flume
and a custom Kafka Sink.
Search for 'Artimon' on slideshare.
On Apr 9, 2012 9:30 AM, "liu brent" <li...@gmail.com> wrote:
> Hi,
> Currently, there are 20+ kinds of web service running on hundreds of
> machines in 4 data centers, which are written in java. We want to collect
> and store the statics of JVM, MBeans of JMX, cpu usage etc of the service
> and machines, and then send the waring to Nagios. Does anyone has the
> experience in using Kafka to do so?
> The direct solution seems like that we install a client on every machine,
> which works as JMX client, collect the information and work as kafka
> producer. Anyone has better ideas or suggestions.
>
> Thanks,
> Liu
>