You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Wollert, Fabian" <fa...@zalando.de> on 2015/12/10 16:14:07 UTC
Per Topic Metrics
Hi everyone, when i browse via jconsole through a freshly started kafka
0.8.2.1 JVM i can only find metrics for the whole broker (like
MessagesInPerSec in
kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics") but not
per topic. what am i missing, e.g. where can i find those metrics?
the documentation (
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-ServerStats)
says kafka:type=kafka.BrokerAllTopicStat.[topic] but this is not exisiting
for me, i guess this is related to 0.7??
Cheers
--
*Fabian Wollert*
Business Intelligence
*Zalando SE*
*POSTAL ADDRESS*
Zalando SE
11501 Berlin
*OFFICE*
Zalando SE
Mollstraße 1
10178 Berlin
Germany
Phone: +49 30 20968 1819
Fax: +49 30 27594 693
E-Mail: fabian.wollert@zalando.de
Web: www.zalando.de
Jobs: jobs.zalando.de
Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
Company registration: Amtsgericht Charlottenburg, HRB 158855 B
Tax ID: 29/560/00596 * VAT registration number: DE 260543043
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of the Supervisory Board: Cristina Stenbeck
Registered office: Berlinn
Re: Per Topic Metrics
Posted by Otis Gospodnetić <ot...@gmail.com>.
Fabian,
Have a look at jmxc - it's a simple command-line tool we open-sourced ages
ago. Very handy for dumping the whole JMX content, which helps when you
don't know what exactly you are looking for :)
https://github.com/sematext/jmxc
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Tue, Dec 15, 2015 at 5:37 AM, Wollert, Fabian <fa...@zalando.de>
wrote:
> i found the problem by myself. since i don't know nothing about JMX and the
> Beans, i was querying the bean wrong. i still don't know how to query the
> specific topic metrics in jconsole. but it works for me to query (with
> jolokia) instead of:
>
> kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics
>
> to query:
>
> kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics, topic=<TOPIC_NAME>
>
> still, this is not like it says in the documentation. maybe worth
> reworking?
>
> Cheers
>
> Fabian
>
> 2015-12-11 21:28 GMT+01:00 Otis Gospodnetić <ot...@gmail.com>:
>
> > Hi Fabian,
> >
> > Hmmm, there are certainly per-topic metrics.
> > You can see it in this SPM demo for Kafka monitoring:
> > * click on https://apps.sematext.com/demo to get into the demo account
> > * go into any SPM app with "kafka" in the name, like
> >
> https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293
> > * click on "Kafka" on the left side
> >
> > You should see per-topic metrics there. For example here:
> >
> >
> https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293&r=kafkaBrokerTopicBytesMassagesFailedReportPage×tamp=1449865640781&stickyFiltersOff=false
> >
> > So you should be able to find all those things in JMX, too. Note that
> some
> > metrics are in Broker, some in Producer, and some in Consumer JMX.
> >
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection
> > Solr & Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> > On Thu, Dec 10, 2015 at 10:14 AM, Wollert, Fabian <
> > fabian.wollert@zalando.de
> > > wrote:
> >
> > > Hi everyone, when i browse via jconsole through a freshly started kafka
> > > 0.8.2.1 JVM i can only find metrics for the whole broker (like
> > > MessagesInPerSec in
> > > kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics") but not
> > > per topic. what am i missing, e.g. where can i find those metrics?
> > >
> > > the documentation (
> > >
> > >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-ServerStats
> > > )
> > > says kafka:type=kafka.BrokerAllTopicStat.[topic] but this is not
> > exisiting
> > > for me, i guess this is related to 0.7??
> > >
> > > Cheers
> > >
> > > --
> > > *Fabian Wollert*
> > > Business Intelligence
> > >
> > > *Zalando SE*
> > >
> > > *POSTAL ADDRESS*
> > > Zalando SE
> > > 11501 Berlin
> > >
> > > *OFFICE*
> > > Zalando SE
> > > Mollstraße 1
> > > 10178 Berlin
> > > Germany
> > >
> > > Phone: +49 30 20968 1819
> > > Fax: +49 30 27594 693
> > > E-Mail: fabian.wollert@zalando.de
> > > Web: www.zalando.de
> > > Jobs: jobs.zalando.de
> > >
> > > Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
> > > Company registration: Amtsgericht Charlottenburg, HRB 158855 B
> > > Tax ID: 29/560/00596 * VAT registration number: DE 260543043
> > > Management Board: Robert Gentz, David Schneider, Rubin Ritter
> > > Chairperson of the Supervisory Board: Cristina Stenbeck
> > > Registered office: Berlinn
> > >
> >
>
>
>
> --
> *Fabian Wollert*
> Business Intelligence
>
> *Zalando SE*
>
> *POSTAL ADDRESS*
> Zalando SE
> 11501 Berlin
>
> *OFFICE*
> Zalando SE
> Mollstraße 1
> 10178 Berlin
> Germany
>
> Phone: +49 30 20968 1819
> Fax: +49 30 27594 693
> E-Mail: fabian.wollert@zalando.de
> Web: www.zalando.de
> Jobs: jobs.zalando.de
>
> Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
> Company registration: Amtsgericht Charlottenburg, HRB 158855 B
> Tax ID: 29/560/00596 * VAT registration number: DE 260543043
> Management Board: Robert Gentz, David Schneider, Rubin Ritter
> Chairperson of the Supervisory Board: Cristina Stenbeck
> Registered office: Berlinn
>
Re: Per Topic Metrics
Posted by "Wollert, Fabian" <fa...@zalando.de>.
i found the problem by myself. since i don't know nothing about JMX and the
Beans, i was querying the bean wrong. i still don't know how to query the
specific topic metrics in jconsole. but it works for me to query (with
jolokia) instead of:
kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics
to query:
kafka.server:name=BytesInPerSec,type=BrokerTopicMetrics, topic=<TOPIC_NAME>
still, this is not like it says in the documentation. maybe worth reworking?
Cheers
Fabian
2015-12-11 21:28 GMT+01:00 Otis Gospodnetić <ot...@gmail.com>:
> Hi Fabian,
>
> Hmmm, there are certainly per-topic metrics.
> You can see it in this SPM demo for Kafka monitoring:
> * click on https://apps.sematext.com/demo to get into the demo account
> * go into any SPM app with "kafka" in the name, like
> https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293
> * click on "Kafka" on the left side
>
> You should see per-topic metrics there. For example here:
>
> https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293&r=kafkaBrokerTopicBytesMassagesFailedReportPage×tamp=1449865640781&stickyFiltersOff=false
>
> So you should be able to find all those things in JMX, too. Note that some
> metrics are in Broker, some in Producer, and some in Consumer JMX.
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection
> Solr & Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Thu, Dec 10, 2015 at 10:14 AM, Wollert, Fabian <
> fabian.wollert@zalando.de
> > wrote:
>
> > Hi everyone, when i browse via jconsole through a freshly started kafka
> > 0.8.2.1 JVM i can only find metrics for the whole broker (like
> > MessagesInPerSec in
> > kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics") but not
> > per topic. what am i missing, e.g. where can i find those metrics?
> >
> > the documentation (
> >
> >
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-ServerStats
> > )
> > says kafka:type=kafka.BrokerAllTopicStat.[topic] but this is not
> exisiting
> > for me, i guess this is related to 0.7??
> >
> > Cheers
> >
> > --
> > *Fabian Wollert*
> > Business Intelligence
> >
> > *Zalando SE*
> >
> > *POSTAL ADDRESS*
> > Zalando SE
> > 11501 Berlin
> >
> > *OFFICE*
> > Zalando SE
> > Mollstraße 1
> > 10178 Berlin
> > Germany
> >
> > Phone: +49 30 20968 1819
> > Fax: +49 30 27594 693
> > E-Mail: fabian.wollert@zalando.de
> > Web: www.zalando.de
> > Jobs: jobs.zalando.de
> >
> > Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
> > Company registration: Amtsgericht Charlottenburg, HRB 158855 B
> > Tax ID: 29/560/00596 * VAT registration number: DE 260543043
> > Management Board: Robert Gentz, David Schneider, Rubin Ritter
> > Chairperson of the Supervisory Board: Cristina Stenbeck
> > Registered office: Berlinn
> >
>
--
*Fabian Wollert*
Business Intelligence
*Zalando SE*
*POSTAL ADDRESS*
Zalando SE
11501 Berlin
*OFFICE*
Zalando SE
Mollstraße 1
10178 Berlin
Germany
Phone: +49 30 20968 1819
Fax: +49 30 27594 693
E-Mail: fabian.wollert@zalando.de
Web: www.zalando.de
Jobs: jobs.zalando.de
Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
Company registration: Amtsgericht Charlottenburg, HRB 158855 B
Tax ID: 29/560/00596 * VAT registration number: DE 260543043
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of the Supervisory Board: Cristina Stenbeck
Registered office: Berlinn
Re: Per Topic Metrics
Posted by Otis Gospodnetić <ot...@gmail.com>.
Hi Fabian,
Hmmm, there are certainly per-topic metrics.
You can see it in this SPM demo for Kafka monitoring:
* click on https://apps.sematext.com/demo to get into the demo account
* go into any SPM app with "kafka" in the name, like
https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293
* click on "Kafka" on the left side
You should see per-topic metrics there. For example here:
https://apps.sematext.com/spm-reports/mainPage.do?selectedApplication=4293&r=kafkaBrokerTopicBytesMassagesFailedReportPage×tamp=1449865640781&stickyFiltersOff=false
So you should be able to find all those things in JMX, too. Note that some
metrics are in Broker, some in Producer, and some in Consumer JMX.
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Thu, Dec 10, 2015 at 10:14 AM, Wollert, Fabian <fabian.wollert@zalando.de
> wrote:
> Hi everyone, when i browse via jconsole through a freshly started kafka
> 0.8.2.1 JVM i can only find metrics for the whole broker (like
> MessagesInPerSec in
> kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics") but not
> per topic. what am i missing, e.g. where can i find those metrics?
>
> the documentation (
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-ServerStats
> )
> says kafka:type=kafka.BrokerAllTopicStat.[topic] but this is not exisiting
> for me, i guess this is related to 0.7??
>
> Cheers
>
> --
> *Fabian Wollert*
> Business Intelligence
>
> *Zalando SE*
>
> *POSTAL ADDRESS*
> Zalando SE
> 11501 Berlin
>
> *OFFICE*
> Zalando SE
> Mollstraße 1
> 10178 Berlin
> Germany
>
> Phone: +49 30 20968 1819
> Fax: +49 30 27594 693
> E-Mail: fabian.wollert@zalando.de
> Web: www.zalando.de
> Jobs: jobs.zalando.de
>
> Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
> Company registration: Amtsgericht Charlottenburg, HRB 158855 B
> Tax ID: 29/560/00596 * VAT registration number: DE 260543043
> Management Board: Robert Gentz, David Schneider, Rubin Ritter
> Chairperson of the Supervisory Board: Cristina Stenbeck
> Registered office: Berlinn
>
RE: Per Topic Metrics
Posted by Todd Snyder <ts...@blackberry.com>.
I want to say that the metrics only show up when the first message comes in, but I could be thinking of another tool.
Try sending a message to the broker and see if metrics appear?
t.
-----Original Message-----
From: Wollert, Fabian [mailto:fabian.wollert@zalando.de]
Sent: Thursday, December 10, 2015 10:14
To: users@kafka.apache.org
Subject: Per Topic Metrics
Hi everyone, when i browse via jconsole through a freshly started kafka
0.8.2.1 JVM i can only find metrics for the whole broker (like
MessagesInPerSec in
kafka.server:name=MessagesInPerSec,type=BrokerTopicMetrics") but not
per topic. what am i missing, e.g. where can i find those metrics?
the documentation (
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-ServerStats)
says kafka:type=kafka.BrokerAllTopicStat.[topic] but this is not exisiting
for me, i guess this is related to 0.7??
Cheers
--
*Fabian Wollert*
Business Intelligence
*Zalando SE*
*POSTAL ADDRESS*
Zalando SE
11501 Berlin
*OFFICE*
Zalando SE
Mollstraße 1
10178 Berlin
Germany
Phone: +49 30 20968 1819
Fax: +49 30 27594 693
E-Mail: fabian.wollert@zalando.de
Web: www.zalando.de
Jobs: jobs.zalando.de
Zalando SE, Tamara-Danz-Straße 1, 10243 Berlin
Company registration: Amtsgericht Charlottenburg, HRB 158855 B
Tax ID: 29/560/00596 * VAT registration number: DE 260543043
Management Board: Robert Gentz, David Schneider, Rubin Ritter
Chairperson of the Supervisory Board: Cristina Stenbeck
Registered office: Berlinn