You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Otis Gospodnetic <ot...@gmail.com> on 2013/07/24 16:29:20 UTC
Metrics: via Broker vs. Producer vs. Consumer
Hi,
I was looking at
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
and noticed there is no information about which metrics are available
in which process/JVM/JMX.
Some are available in the Broker process, but some are only available
from the JVM running Consumer and some only from the JVM running
Producer. And yet some Producer and Consumer metrics are, I *believe*
available from Broker's JMX.
Would it be possible for somebody in the know to mark the metrics in
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
so one can tell where to get it?
Also, why is it that the Broker process doesn't have *all* metrics,
including Producer and Consumer one? Is that because there can be N
Brokers and each P or C talk to one Broker at a time and thus there is
no single process/JMX that can know *all* stats for *all* Brokers and
for *all* Ps and Cs?
Thank you!
Otis
--
Performance Monitoring -- http://sematext.com/spm
Solr & ElasticSearch Support -- http://sematext.com/
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Scott Clasen <sc...@heroku.com>.
Valid assumptions IMO
Sent from my iPhone
On Jul 24, 2013, at 8:22 AM, Jay Kreps <ja...@gmail.com> wrote:
> Yeah all our monitoring is just of the local process (basically just a
> counter exposed through yammer metrics which support jmx and other
> outputs). If I understand what you want instead of having a counter that
> tracks, say, produce requests per second for a single broker you want one
> that covers the whole cluster. Obviously this would require collecting the
> local count and aggregating across all the brokers.
>
> Our assumption is that you already have a separate monitoring system which
> can slurp all these up, aggregate them, graph them, and alert off them.
> There are a number of open source thingies like this and I think most
> bigger shops have something they use. Our assumption is that trying to do a
> kafka-specific monitoring system wouldn't work for most people because they
> are wedded to their current setup and just want to integrate with that.
>
> I'm not sure how valid any of those assumptions actually are.
>
> -Jay
>
>
> On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com> wrote:
>
>> Hi,
>>
>> I was looking at
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> and noticed there is no information about which metrics are available
>> in which process/JVM/JMX.
>>
>> Some are available in the Broker process, but some are only available
>> from the JVM running Consumer and some only from the JVM running
>> Producer. And yet some Producer and Consumer metrics are, I *believe*
>> available from Broker's JMX.
>>
>> Would it be possible for somebody in the know to mark the metrics in
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> so one can tell where to get it?
>>
>> Also, why is it that the Broker process doesn't have *all* metrics,
>> including Producer and Consumer one? Is that because there can be N
>> Brokers and each P or C talk to one Broker at a time and thus there is
>> no single process/JMX that can know *all* stats for *all* Brokers and
>> for *all* Ps and Cs?
>>
>> Thank you!
>> Otis
>> --
>> Performance Monitoring -- http://sematext.com/spm
>> Solr & ElasticSearch Support -- http://sematext.com/
>>
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Jun Rao <ju...@gmail.com>.
On the broker side, we have jmx beans for producer/consumer request rate
and time from all clients. Each producer/consumer client has jmx beans that
tracks its own request rate and time.
Thanks,
Jun
On Thu, Jul 25, 2013 at 2:35 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Hi,
>
> Oh, I know about Yammer metrics, Ganglia, Graphite, and friends....
>
> My main Q was: can I get all Consumer and all Producer stats via Broker's
> JMX. I think the answer is no :(
> So I'm forced to look at the JMX of individual applications that act as
> Kafka Consumers or Producers.
>
> Btw. this is for SPM - http://sematext.com/spm - support for monitoring
> Kafka should be in the early August release.
>
> As a matter of fact, there is an SPM Reporter from Yammer metrics, so
> people should be able to get their Kafka 0.8 metrics graphed as custom
> metrics in SPM already.
>
> Thanks for the patience and the info, Jay!
>
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>
>
> >________________________________
> > From: Jay Kreps <ja...@gmail.com>
> >To: "users@kafka.apache.org" <us...@kafka.apache.org>; Otis Gospodnetic <
> otis_gospodnetic@yahoo.com>
> >Sent: Thursday, July 25, 2013 1:07 AM
> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
> >
> >
> >Yeah totally that was what I thought you are saying. So what we have is a
> >yammer metrics which allows you to plug in any metrics reporter (it will
> do
> >jmx or whatever). But all this does it let you customize how the
> individual
> >java processes output metrics. We don't solve the problem of putting all
> >these together into a central dashboard. But these can be hooked up to a
> >standard monitoring thingy that pulls in metrics from all the java
> >processes in your environment (brokers, consumers, non-kafka related
> >things, etc) and let's you see these stats in aggregate across all
> machines.
> >
> >I'm actually kind of ignorant of the state of open source monitoring
> >thingies because at linkedin we have a pretty badass in-house thing that
> >does this and is used for everything. I guess a lot of people used to use
> >ganglia or nagios for this but nowadays I hear all the cool kids use
> >graphite?
> >
> >At various times we have tossed around the idea of having a "Kafka
> >Monitoring Dashboard" that had a lot of stats on it. This would allow us
> to
> >make out-of-the-box monitoring a lot easier for Kafka. The feedback we
> have
> >gotten previously is that everyone has some inhouse tool they already use
> >for graphing/alerting/etc and they don't really want a one-off solution
> for
> >Kafka. This is definitely true for our own usage.
> >
> >-Jay
> >
> >
> >
> >On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic <
> >otis_gospodnetic@yahoo.com> wrote:
> >
> >> Hi,
> >>
> >> Ah, I think I didn't ask my question clearly. Another try:
> >> * If I have a javaagent attached to the Kafka process, I'll be able to
> >> connect to its JMX and get all the Broker metrics for that Broker
> process.
> >> * If I have another Broker process, I'll need to attach my agent to this
> >> process, too, to get all Broker metrics associated with this second
> Broker
> >> process.
> >>
> >>
> >> So far OK - like you said, I can sum, average, etc.
> >>
> >> But what if I want to get all Producer metrics? What do I need to do? I
> >> *believe* I would have to attach the javaagent to whichever app is
> acting
> >> as a Kafka Consumer and get Consumer stats from the JMX associated with
> the
> >> JVM process running that app.
> >> Is this correct?
> >> Is there any way to avoid that and get all Consumer and all Producer
> >> metrics using the javaagent attached to one of the Broker processes?
> >>
> >> Thanks,
> >> Otis
> >> ----
> >> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> >> http://sematext.com/spm
> >>
> >>
> >>
> >>
> >> >________________________________
> >> > From: Jay Kreps <ja...@gmail.com>
> >> >To: "users@kafka.apache.org" <us...@kafka.apache.org>
> >> >Sent: Wednesday, July 24, 2013 5:22 PM
> >> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
> >> >
> >> >
> >> >Yeah all our monitoring is just of the local process (basically just a
> >> >counter exposed through yammer metrics which support jmx and other
> >> >outputs). If I understand what you want instead of having a counter
> that
> >> >tracks, say, produce requests per second for a single broker you want
> one
> >> >that covers the whole cluster. Obviously this would require collecting
> the
> >> >local count and aggregating across all the brokers.
> >> >
> >> >Our assumption is that you already have a separate monitoring system
> which
> >> >can slurp all these up, aggregate them, graph them, and alert off them.
> >> >There are a number of open source thingies like this and I think most
> >> >bigger shops have something they use. Our assumption is that trying to
> do
> >> a
> >> >kafka-specific monitoring system wouldn't work for most people because
> >> they
> >> >are wedded to their current setup and just want to integrate with that.
> >> >
> >> >I'm not sure how valid any of those assumptions actually are.
> >> >
> >> >-Jay
> >> >
> >> >
> >> >On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
> >> >otis.gospodnetic@gmail.com> wrote:
> >> >
> >> >> Hi,
> >> >>
> >> >> I was looking at
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> >> >> and noticed there is no information about which metrics are available
> >> >> in which process/JVM/JMX.
> >> >>
> >> >> Some are available in the Broker process, but some are only available
> >> >> from the JVM running Consumer and some only from the JVM running
> >> >> Producer. And yet some Producer and Consumer metrics are, I
> *believe*
> >> >> available from Broker's JMX.
> >> >>
> >> >> Would it be possible for somebody in the know to mark the metrics in
> >> >>
> >> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> >> >> so one can tell where to get it?
> >> >>
> >> >> Also, why is it that the Broker process doesn't have *all* metrics,
> >> >> including Producer and Consumer one? Is that because there can be N
> >> >> Brokers and each P or C talk to one Broker at a time and thus there
> is
> >> >> no single process/JMX that can know *all* stats for *all* Brokers and
> >> >> for *all* Ps and Cs?
> >> >>
> >> >> Thank you!
> >> >> Otis
> >> >> --
> >> >> Performance Monitoring -- http://sematext.com/spm
> >> >> Solr & ElasticSearch Support -- http://sematext.com/
> >> >>
> >> >
> >> >
> >> >
> >>
> >
> >
> >
>
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Oh, I know about Yammer metrics, Ganglia, Graphite, and friends....
My main Q was: can I get all Consumer and all Producer stats via Broker's JMX. I think the answer is no :(
So I'm forced to look at the JMX of individual applications that act as Kafka Consumers or Producers.
Btw. this is for SPM - http://sematext.com/spm - support for monitoring Kafka should be in the early August release.
As a matter of fact, there is an SPM Reporter from Yammer metrics, so people should be able to get their Kafka 0.8 metrics graphed as custom metrics in SPM already.
Thanks for the patience and the info, Jay!
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop - http://sematext.com/spm
>________________________________
> From: Jay Kreps <ja...@gmail.com>
>To: "users@kafka.apache.org" <us...@kafka.apache.org>; Otis Gospodnetic <ot...@yahoo.com>
>Sent: Thursday, July 25, 2013 1:07 AM
>Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
>
>
>Yeah totally that was what I thought you are saying. So what we have is a
>yammer metrics which allows you to plug in any metrics reporter (it will do
>jmx or whatever). But all this does it let you customize how the individual
>java processes output metrics. We don't solve the problem of putting all
>these together into a central dashboard. But these can be hooked up to a
>standard monitoring thingy that pulls in metrics from all the java
>processes in your environment (brokers, consumers, non-kafka related
>things, etc) and let's you see these stats in aggregate across all machines.
>
>I'm actually kind of ignorant of the state of open source monitoring
>thingies because at linkedin we have a pretty badass in-house thing that
>does this and is used for everything. I guess a lot of people used to use
>ganglia or nagios for this but nowadays I hear all the cool kids use
>graphite?
>
>At various times we have tossed around the idea of having a "Kafka
>Monitoring Dashboard" that had a lot of stats on it. This would allow us to
>make out-of-the-box monitoring a lot easier for Kafka. The feedback we have
>gotten previously is that everyone has some inhouse tool they already use
>for graphing/alerting/etc and they don't really want a one-off solution for
>Kafka. This is definitely true for our own usage.
>
>-Jay
>
>
>
>On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic <
>otis_gospodnetic@yahoo.com> wrote:
>
>> Hi,
>>
>> Ah, I think I didn't ask my question clearly. Another try:
>> * If I have a javaagent attached to the Kafka process, I'll be able to
>> connect to its JMX and get all the Broker metrics for that Broker process.
>> * If I have another Broker process, I'll need to attach my agent to this
>> process, too, to get all Broker metrics associated with this second Broker
>> process.
>>
>>
>> So far OK - like you said, I can sum, average, etc.
>>
>> But what if I want to get all Producer metrics? What do I need to do? I
>> *believe* I would have to attach the javaagent to whichever app is acting
>> as a Kafka Consumer and get Consumer stats from the JMX associated with the
>> JVM process running that app.
>> Is this correct?
>> Is there any way to avoid that and get all Consumer and all Producer
>> metrics using the javaagent attached to one of the Broker processes?
>>
>> Thanks,
>> Otis
>> ----
>> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
>> http://sematext.com/spm
>>
>>
>>
>>
>> >________________________________
>> > From: Jay Kreps <ja...@gmail.com>
>> >To: "users@kafka.apache.org" <us...@kafka.apache.org>
>> >Sent: Wednesday, July 24, 2013 5:22 PM
>> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
>> >
>> >
>> >Yeah all our monitoring is just of the local process (basically just a
>> >counter exposed through yammer metrics which support jmx and other
>> >outputs). If I understand what you want instead of having a counter that
>> >tracks, say, produce requests per second for a single broker you want one
>> >that covers the whole cluster. Obviously this would require collecting the
>> >local count and aggregating across all the brokers.
>> >
>> >Our assumption is that you already have a separate monitoring system which
>> >can slurp all these up, aggregate them, graph them, and alert off them.
>> >There are a number of open source thingies like this and I think most
>> >bigger shops have something they use. Our assumption is that trying to do
>> a
>> >kafka-specific monitoring system wouldn't work for most people because
>> they
>> >are wedded to their current setup and just want to integrate with that.
>> >
>> >I'm not sure how valid any of those assumptions actually are.
>> >
>> >-Jay
>> >
>> >
>> >On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
>> >otis.gospodnetic@gmail.com> wrote:
>> >
>> >> Hi,
>> >>
>> >> I was looking at
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> >> and noticed there is no information about which metrics are available
>> >> in which process/JVM/JMX.
>> >>
>> >> Some are available in the Broker process, but some are only available
>> >> from the JVM running Consumer and some only from the JVM running
>> >> Producer. And yet some Producer and Consumer metrics are, I *believe*
>> >> available from Broker's JMX.
>> >>
>> >> Would it be possible for somebody in the know to mark the metrics in
>> >>
>> >>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> >> so one can tell where to get it?
>> >>
>> >> Also, why is it that the Broker process doesn't have *all* metrics,
>> >> including Producer and Consumer one? Is that because there can be N
>> >> Brokers and each P or C talk to one Broker at a time and thus there is
>> >> no single process/JMX that can know *all* stats for *all* Brokers and
>> >> for *all* Ps and Cs?
>> >>
>> >> Thank you!
>> >> Otis
>> >> --
>> >> Performance Monitoring -- http://sematext.com/spm
>> >> Solr & ElasticSearch Support -- http://sematext.com/
>> >>
>> >
>> >
>> >
>>
>
>
>
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Jay Kreps <ja...@gmail.com>.
Yeah totally that was what I thought you are saying. So what we have is a
yammer metrics which allows you to plug in any metrics reporter (it will do
jmx or whatever). But all this does it let you customize how the individual
java processes output metrics. We don't solve the problem of putting all
these together into a central dashboard. But these can be hooked up to a
standard monitoring thingy that pulls in metrics from all the java
processes in your environment (brokers, consumers, non-kafka related
things, etc) and let's you see these stats in aggregate across all machines.
I'm actually kind of ignorant of the state of open source monitoring
thingies because at linkedin we have a pretty badass in-house thing that
does this and is used for everything. I guess a lot of people used to use
ganglia or nagios for this but nowadays I hear all the cool kids use
graphite?
At various times we have tossed around the idea of having a "Kafka
Monitoring Dashboard" that had a lot of stats on it. This would allow us to
make out-of-the-box monitoring a lot easier for Kafka. The feedback we have
gotten previously is that everyone has some inhouse tool they already use
for graphing/alerting/etc and they don't really want a one-off solution for
Kafka. This is definitely true for our own usage.
-Jay
On Wed, Jul 24, 2013 at 10:04 AM, Otis Gospodnetic <
otis_gospodnetic@yahoo.com> wrote:
> Hi,
>
> Ah, I think I didn't ask my question clearly. Another try:
> * If I have a javaagent attached to the Kafka process, I'll be able to
> connect to its JMX and get all the Broker metrics for that Broker process.
> * If I have another Broker process, I'll need to attach my agent to this
> process, too, to get all Broker metrics associated with this second Broker
> process.
>
>
> So far OK - like you said, I can sum, average, etc.
>
> But what if I want to get all Producer metrics? What do I need to do? I
> *believe* I would have to attach the javaagent to whichever app is acting
> as a Kafka Consumer and get Consumer stats from the JMX associated with the
> JVM process running that app.
> Is this correct?
> Is there any way to avoid that and get all Consumer and all Producer
> metrics using the javaagent attached to one of the Broker processes?
>
> Thanks,
> Otis
> ----
> Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop -
> http://sematext.com/spm
>
>
>
>
> >________________________________
> > From: Jay Kreps <ja...@gmail.com>
> >To: "users@kafka.apache.org" <us...@kafka.apache.org>
> >Sent: Wednesday, July 24, 2013 5:22 PM
> >Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
> >
> >
> >Yeah all our monitoring is just of the local process (basically just a
> >counter exposed through yammer metrics which support jmx and other
> >outputs). If I understand what you want instead of having a counter that
> >tracks, say, produce requests per second for a single broker you want one
> >that covers the whole cluster. Obviously this would require collecting the
> >local count and aggregating across all the brokers.
> >
> >Our assumption is that you already have a separate monitoring system which
> >can slurp all these up, aggregate them, graph them, and alert off them.
> >There are a number of open source thingies like this and I think most
> >bigger shops have something they use. Our assumption is that trying to do
> a
> >kafka-specific monitoring system wouldn't work for most people because
> they
> >are wedded to their current setup and just want to integrate with that.
> >
> >I'm not sure how valid any of those assumptions actually are.
> >
> >-Jay
> >
> >
> >On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
> >otis.gospodnetic@gmail.com> wrote:
> >
> >> Hi,
> >>
> >> I was looking at
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> >> and noticed there is no information about which metrics are available
> >> in which process/JVM/JMX.
> >>
> >> Some are available in the Broker process, but some are only available
> >> from the JVM running Consumer and some only from the JVM running
> >> Producer. And yet some Producer and Consumer metrics are, I *believe*
> >> available from Broker's JMX.
> >>
> >> Would it be possible for somebody in the know to mark the metrics in
> >>
> >>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> >> so one can tell where to get it?
> >>
> >> Also, why is it that the Broker process doesn't have *all* metrics,
> >> including Producer and Consumer one? Is that because there can be N
> >> Brokers and each P or C talk to one Broker at a time and thus there is
> >> no single process/JMX that can know *all* stats for *all* Brokers and
> >> for *all* Ps and Cs?
> >>
> >> Thank you!
> >> Otis
> >> --
> >> Performance Monitoring -- http://sematext.com/spm
> >> Solr & ElasticSearch Support -- http://sematext.com/
> >>
> >
> >
> >
>
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Otis Gospodnetic <ot...@yahoo.com>.
Hi,
Ah, I think I didn't ask my question clearly. Another try:
* If I have a javaagent attached to the Kafka process, I'll be able to connect to its JMX and get all the Broker metrics for that Broker process.
* If I have another Broker process, I'll need to attach my agent to this process, too, to get all Broker metrics associated with this second Broker process.
So far OK - like you said, I can sum, average, etc.
But what if I want to get all Producer metrics? What do I need to do? I *believe* I would have to attach the javaagent to whichever app is acting as a Kafka Consumer and get Consumer stats from the JMX associated with the JVM process running that app.
Is this correct?
Is there any way to avoid that and get all Consumer and all Producer metrics using the javaagent attached to one of the Broker processes?
Thanks,
Otis
----
Performance Monitoring for Solr / ElasticSearch / HBase / Hadoop - http://sematext.com/spm
>________________________________
> From: Jay Kreps <ja...@gmail.com>
>To: "users@kafka.apache.org" <us...@kafka.apache.org>
>Sent: Wednesday, July 24, 2013 5:22 PM
>Subject: Re: Metrics: via Broker vs. Producer vs. Consumer
>
>
>Yeah all our monitoring is just of the local process (basically just a
>counter exposed through yammer metrics which support jmx and other
>outputs). If I understand what you want instead of having a counter that
>tracks, say, produce requests per second for a single broker you want one
>that covers the whole cluster. Obviously this would require collecting the
>local count and aggregating across all the brokers.
>
>Our assumption is that you already have a separate monitoring system which
>can slurp all these up, aggregate them, graph them, and alert off them.
>There are a number of open source thingies like this and I think most
>bigger shops have something they use. Our assumption is that trying to do a
>kafka-specific monitoring system wouldn't work for most people because they
>are wedded to their current setup and just want to integrate with that.
>
>I'm not sure how valid any of those assumptions actually are.
>
>-Jay
>
>
>On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
>otis.gospodnetic@gmail.com> wrote:
>
>> Hi,
>>
>> I was looking at
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> and noticed there is no information about which metrics are available
>> in which process/JVM/JMX.
>>
>> Some are available in the Broker process, but some are only available
>> from the JVM running Consumer and some only from the JVM running
>> Producer. And yet some Producer and Consumer metrics are, I *believe*
>> available from Broker's JMX.
>>
>> Would it be possible for somebody in the know to mark the metrics in
>>
>> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
>> so one can tell where to get it?
>>
>> Also, why is it that the Broker process doesn't have *all* metrics,
>> including Producer and Consumer one? Is that because there can be N
>> Brokers and each P or C talk to one Broker at a time and thus there is
>> no single process/JMX that can know *all* stats for *all* Brokers and
>> for *all* Ps and Cs?
>>
>> Thank you!
>> Otis
>> --
>> Performance Monitoring -- http://sematext.com/spm
>> Solr & ElasticSearch Support -- http://sematext.com/
>>
>
>
>
Re: Metrics: via Broker vs. Producer vs. Consumer
Posted by Jay Kreps <ja...@gmail.com>.
Yeah all our monitoring is just of the local process (basically just a
counter exposed through yammer metrics which support jmx and other
outputs). If I understand what you want instead of having a counter that
tracks, say, produce requests per second for a single broker you want one
that covers the whole cluster. Obviously this would require collecting the
local count and aggregating across all the brokers.
Our assumption is that you already have a separate monitoring system which
can slurp all these up, aggregate them, graph them, and alert off them.
There are a number of open source thingies like this and I think most
bigger shops have something they use. Our assumption is that trying to do a
kafka-specific monitoring system wouldn't work for most people because they
are wedded to their current setup and just want to integrate with that.
I'm not sure how valid any of those assumptions actually are.
-Jay
On Wed, Jul 24, 2013 at 7:29 AM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> I was looking at
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> and noticed there is no information about which metrics are available
> in which process/JVM/JMX.
>
> Some are available in the Broker process, but some are only available
> from the JVM running Consumer and some only from the JVM running
> Producer. And yet some Producer and Consumer metrics are, I *believe*
> available from Broker's JMX.
>
> Would it be possible for somebody in the know to mark the metrics in
>
> https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
> so one can tell where to get it?
>
> Also, why is it that the Broker process doesn't have *all* metrics,
> including Producer and Consumer one? Is that because there can be N
> Brokers and each P or C talk to one Broker at a time and thus there is
> no single process/JMX that can know *all* stats for *all* Brokers and
> for *all* Ps and Cs?
>
> Thank you!
> Otis
> --
> Performance Monitoring -- http://sematext.com/spm
> Solr & ElasticSearch Support -- http://sematext.com/
>