You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Saladi Naidu <na...@yahoo.com.INVALID> on 2015/04/20 21:00:28 UTC
Kafka Monitoring using JMX
We are planning to monitor health of our Kafka environment using JMX. I have looked at below links to find what is available via
https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
why some of the kafka objects ("kafka.cluster", "kafka.consumer" etc are written with an additional " ?
E.g "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" Vs the normal Mbeans are just something like this:
java.lang:type=GarbageCollector,name=ConcurrentMarkSweep
We are not successful in calling objects that have "" in their names. Naidu Saladi
Re: Kafka Monitoring using JMX
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
Not sure what you are using to get the data out, but if you use SPM
<http://sematext.com/spm/integrations/kafka-monitoring.html> you don't have
to worry about any of that... assuming you use either 0.7.x or 0.8.2.x.
Kafka metrics have changed drastically in the past and it looks like more
changes are ahead, so whatever custom you use now will need to be changed
when you update to Kafka with new metrics (format).
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Apr 20, 2015 at 3:00 PM, Saladi Naidu <naidusp2002@yahoo.com.invalid
> wrote:
> We are planning to monitor health of our Kafka environment using JMX. I
> have looked at below links to find what is available via
>
> https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics
> https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
>
> why some of the kafka objects ("kafka.cluster", "kafka.consumer" etc are
> written with an additional " ?
>
> E.g "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" Vs
> the normal Mbeans are just something like this:
>
> java.lang:type=GarbageCollector,name=ConcurrentMarkSweep
>
> We are not successful in calling objects that have "" in their
> names. Naidu Saladi
>
Re: Kafka Monitoring using JMX
Posted by Bhavesh Mistry <mi...@gmail.com>.
You can use this
https://github.com/Stackdriver/jmxtrans-config-stackdriver/blob/master/jmxtrans/stackdriver/json-specify-instance/kafka.json
as an example of how Mbean are named and how " is being escaped with \",
and just use different output writer for any thing prior to 0.8.2.1
version. After 0.8.2.1, you will have to remove " and there are some
changes type,name etc. Please Google search there is JMXTrans tools to
auto generate JMX trans configuration from the runnning brokers.
Thanks,
Bhavesh
On Mon, Apr 20, 2015 at 1:57 PM, Daniel Compton <
daniel.compton.lists@gmail.com> wrote:
> Hi Naidu
>
> You'll need to escape the " with a \ in the Mbean names. I've run across
> this too and it was al pain. It can get a bit tricky if you're doing it in
> code because you need to account for double escapes and so forth. This is a
> bug in the version of Metrics that Kafka is using. There is a JIRA ticket
> to upgrade it somewhere, but I can't find it right now.
>
> Daniel.
> On Tue, 21 Apr 2015 at 7:02 am Saladi Naidu <naidusp2002@yahoo.com.invalid
> >
> wrote:
>
> > We are planning to monitor health of our Kafka environment using JMX. I
> > have looked at below links to find what is available via
> >
> > https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics
> > https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
> >
> > why some of the kafka objects ("kafka.cluster", "kafka.consumer" etc are
> > written with an additional " ?
> >
> > E.g
> "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" Vs
> > the normal Mbeans are just something like this:
> >
> > java.lang:type=GarbageCollector,name=ConcurrentMarkSweep
> >
> > We are not successful in calling objects that have "" in their
> > names. Naidu Saladi
> >
>
Re: Kafka Monitoring using JMX
Posted by Daniel Compton <da...@gmail.com>.
Hi Naidu
You'll need to escape the " with a \ in the Mbean names. I've run across
this too and it was al pain. It can get a bit tricky if you're doing it in
code because you need to account for double escapes and so forth. This is a
bug in the version of Metrics that Kafka is using. There is a JIRA ticket
to upgrade it somewhere, but I can't find it right now.
Daniel.
On Tue, 21 Apr 2015 at 7:02 am Saladi Naidu <na...@yahoo.com.invalid>
wrote:
> We are planning to monitor health of our Kafka environment using JMX. I
> have looked at below links to find what is available via
>
> https://cwiki.apache.org/confluence/display/KAFKA/Available+Metrics
> https://cwiki.apache.org/confluence/display/KAFKA/JMX+Reporters
>
> why some of the kafka objects ("kafka.cluster", "kafka.consumer" etc are
> written with an additional " ?
>
> E.g "kafka.server":type="BrokerTopicMetrics",name="AllTopicsBytesInPerSec" Vs
> the normal Mbeans are just something like this:
>
> java.lang:type=GarbageCollector,name=ConcurrentMarkSweep
>
> We are not successful in calling objects that have "" in their
> names. Naidu Saladi
>