You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2014/05/28 20:41:21 UTC
Producer Side Metric Details
Hi Kafka Group,
I need to get following metrics from the producer side.
I am able to get following metric using the ProducerTopicMetrics class per
minute.
messageRate
byteRate
droppedMessageRate
I would like to know how to get above metric per topic per partition.
Also, how do I get count of number of exception occurred due to
kafka.common.MessageSizeTooLargeException, or Network
Down or message rejected due to Broker being down etc. Is there a JMX or
any other bean to get this metric counter ?
We are using Kafka version 0.8.0
Thanks,
Bhavesh
Re: Producer Side Metric Details
Posted by Jun Rao <ju...@gmail.com>.
Do you see a jmx per client id?
Thanks,
Jun
On Thu, May 29, 2014 at 9:30 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> There seems to be bug in reporting that is what I did. I registered each
> producer with different client id.
>
> Thanks,
> Bhavesh
>
>
> On Thu, May 29, 2014 at 8:36 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > You need to configure each producer with a different client.id.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 29, 2014 at 5:38 PM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com>
> > wrote:
> >
> > > We have 4 producers (for 4 topics) within one JVM and all of them
> > > configured with same broker list. Each one of the producer has
> > client.id
> > > configured as topic name. When I see the JMX console, I see that "
> > > *ProducerRequestMetric*" JMX have issue, it takes the last client.id
> > > config
> > > and report the metrics.
> > >
> > > 1) Does it really report the Request rate per producer or all the
> > > producers to all the brokers (*AllBrokersProducerRequestRateAndTimeMs
> or
> > > AllBrokersProducerRequestSize *) Is it like global aggregated Request
> > > rate and size across all topics for All Brokers ?
> > >
> > > 2) Similarly under "*ProducerTopicMetrics*", AllTopicsBytesPerSec ,
> > > AllTopicsDroppedMessagesPerSec, AllTopicsMessagesPerSec are per topic
> or
> > > across all topics ? The numbers for AllTopics* are same as per topoic
> > > (logmon.error-*) ? Can you guys please clarify this ? Do you guys
> > > intended to work as global aggregated for ALL topic ?
> > >
> > > Please see the following image:
> > >
> > >
> > >
> > >
> > >
> > > Thanks,
> > > Bhavesh
> > >
> > >
> > > On Wed, May 28, 2014 at 9:44 PM, Jun Rao <ju...@gmail.com> wrote:
> > >
> > > > The producer has a "failedToSend" jmx, which includes all kinds of
> > > errors.
> > > > We don't have the message rate per partition.
> > > >
> > > > Thanks,
> > > >
> > > > Jun
> > > >
> > > >
> > > > On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <
> > > > mistry.p.bhavesh@gmail.com
> > > > > wrote:
> > > >
> > > > > Hi Kafka Group,
> > > > >
> > > > > I need to get following metrics from the producer side.
> > > > >
> > > > > I am able to get following metric using the ProducerTopicMetrics
> > class
> > > > per
> > > > > minute.
> > > > > messageRate
> > > > > byteRate
> > > > > droppedMessageRate
> > > > >
> > > > > I would like to know how to get above metric per topic per
> partition.
> > > > > Also, how do I get count of number of exception occurred due to
> > > > > kafka.common.MessageSizeTooLargeException, or Network
> > > > > Down or message rejected due to Broker being down etc. Is there
> a
> > > JMX
> > > > or
> > > > > any other bean to get this metric counter ?
> > > > >
> > > > > We are using Kafka version 0.8.0
> > > > >
> > > > > Thanks,
> > > > >
> > > > > Bhavesh
> > > > >
> > > >
> > >
> >
>
Re: Producer Side Metric Details
Posted by Bhavesh Mistry <mi...@gmail.com>.
There seems to be bug in reporting that is what I did. I registered each
producer with different client id.
Thanks,
Bhavesh
On Thu, May 29, 2014 at 8:36 PM, Jun Rao <ju...@gmail.com> wrote:
> You need to configure each producer with a different client.id.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 29, 2014 at 5:38 PM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com>
> wrote:
>
> > We have 4 producers (for 4 topics) within one JVM and all of them
> > configured with same broker list. Each one of the producer has
> client.id
> > configured as topic name. When I see the JMX console, I see that "
> > *ProducerRequestMetric*" JMX have issue, it takes the last client.id
> > config
> > and report the metrics.
> >
> > 1) Does it really report the Request rate per producer or all the
> > producers to all the brokers (*AllBrokersProducerRequestRateAndTimeMs or
> > AllBrokersProducerRequestSize *) Is it like global aggregated Request
> > rate and size across all topics for All Brokers ?
> >
> > 2) Similarly under "*ProducerTopicMetrics*", AllTopicsBytesPerSec ,
> > AllTopicsDroppedMessagesPerSec, AllTopicsMessagesPerSec are per topic or
> > across all topics ? The numbers for AllTopics* are same as per topoic
> > (logmon.error-*) ? Can you guys please clarify this ? Do you guys
> > intended to work as global aggregated for ALL topic ?
> >
> > Please see the following image:
> >
> >
> >
> >
> >
> > Thanks,
> > Bhavesh
> >
> >
> > On Wed, May 28, 2014 at 9:44 PM, Jun Rao <ju...@gmail.com> wrote:
> >
> > > The producer has a "failedToSend" jmx, which includes all kinds of
> > errors.
> > > We don't have the message rate per partition.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <
> > > mistry.p.bhavesh@gmail.com
> > > > wrote:
> > >
> > > > Hi Kafka Group,
> > > >
> > > > I need to get following metrics from the producer side.
> > > >
> > > > I am able to get following metric using the ProducerTopicMetrics
> class
> > > per
> > > > minute.
> > > > messageRate
> > > > byteRate
> > > > droppedMessageRate
> > > >
> > > > I would like to know how to get above metric per topic per partition.
> > > > Also, how do I get count of number of exception occurred due to
> > > > kafka.common.MessageSizeTooLargeException, or Network
> > > > Down or message rejected due to Broker being down etc. Is there a
> > JMX
> > > or
> > > > any other bean to get this metric counter ?
> > > >
> > > > We are using Kafka version 0.8.0
> > > >
> > > > Thanks,
> > > >
> > > > Bhavesh
> > > >
> > >
> >
>
Re: Producer Side Metric Details
Posted by Jun Rao <ju...@gmail.com>.
You need to configure each producer with a different client.id.
Thanks,
Jun
On Thu, May 29, 2014 at 5:38 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> We have 4 producers (for 4 topics) within one JVM and all of them
> configured with same broker list. Each one of the producer has client.id
> configured as topic name. When I see the JMX console, I see that "
> *ProducerRequestMetric*" JMX have issue, it takes the last client.id
> config
> and report the metrics.
>
> 1) Does it really report the Request rate per producer or all the
> producers to all the brokers (*AllBrokersProducerRequestRateAndTimeMs or
> AllBrokersProducerRequestSize *) Is it like global aggregated Request
> rate and size across all topics for All Brokers ?
>
> 2) Similarly under "*ProducerTopicMetrics*", AllTopicsBytesPerSec ,
> AllTopicsDroppedMessagesPerSec, AllTopicsMessagesPerSec are per topic or
> across all topics ? The numbers for AllTopics* are same as per topoic
> (logmon.error-*) ? Can you guys please clarify this ? Do you guys
> intended to work as global aggregated for ALL topic ?
>
> Please see the following image:
>
>
>
>
>
> Thanks,
> Bhavesh
>
>
> On Wed, May 28, 2014 at 9:44 PM, Jun Rao <ju...@gmail.com> wrote:
>
> > The producer has a "failedToSend" jmx, which includes all kinds of
> errors.
> > We don't have the message rate per partition.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <
> > mistry.p.bhavesh@gmail.com
> > > wrote:
> >
> > > Hi Kafka Group,
> > >
> > > I need to get following metrics from the producer side.
> > >
> > > I am able to get following metric using the ProducerTopicMetrics class
> > per
> > > minute.
> > > messageRate
> > > byteRate
> > > droppedMessageRate
> > >
> > > I would like to know how to get above metric per topic per partition.
> > > Also, how do I get count of number of exception occurred due to
> > > kafka.common.MessageSizeTooLargeException, or Network
> > > Down or message rejected due to Broker being down etc. Is there a
> JMX
> > or
> > > any other bean to get this metric counter ?
> > >
> > > We are using Kafka version 0.8.0
> > >
> > > Thanks,
> > >
> > > Bhavesh
> > >
> >
>
Re: Producer Side Metric Details
Posted by Bhavesh Mistry <mi...@gmail.com>.
We have 4 producers (for 4 topics) within one JVM and all of them
configured with same broker list. Each one of the producer has client.id
configured as topic name. When I see the JMX console, I see that "
*ProducerRequestMetric*" JMX have issue, it takes the last client.id config
and report the metrics.
1) Does it really report the Request rate per producer or all the
producers to all the brokers (*AllBrokersProducerRequestRateAndTimeMs or
AllBrokersProducerRequestSize *) Is it like global aggregated Request
rate and size across all topics for All Brokers ?
2) Similarly under "*ProducerTopicMetrics*", AllTopicsBytesPerSec ,
AllTopicsDroppedMessagesPerSec, AllTopicsMessagesPerSec are per topic or
across all topics ? The numbers for AllTopics* are same as per topoic
(logmon.error-*) ? Can you guys please clarify this ? Do you guys
intended to work as global aggregated for ALL topic ?
Please see the following image:
Thanks,
Bhavesh
On Wed, May 28, 2014 at 9:44 PM, Jun Rao <ju...@gmail.com> wrote:
> The producer has a "failedToSend" jmx, which includes all kinds of errors.
> We don't have the message rate per partition.
>
> Thanks,
>
> Jun
>
>
> On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > wrote:
>
> > Hi Kafka Group,
> >
> > I need to get following metrics from the producer side.
> >
> > I am able to get following metric using the ProducerTopicMetrics class
> per
> > minute.
> > messageRate
> > byteRate
> > droppedMessageRate
> >
> > I would like to know how to get above metric per topic per partition.
> > Also, how do I get count of number of exception occurred due to
> > kafka.common.MessageSizeTooLargeException, or Network
> > Down or message rejected due to Broker being down etc. Is there a JMX
> or
> > any other bean to get this metric counter ?
> >
> > We are using Kafka version 0.8.0
> >
> > Thanks,
> >
> > Bhavesh
> >
>
Re: Producer Side Metric Details
Posted by Bhavesh Mistry <mi...@gmail.com>.
We have 4 producers (for 4 topics) within one JVM and all of them
configured with same broker list. Each one of the producer has client.id
configured as topic name. When I see the JMX console, I see that "
*ProducerRequestMetric*" JMX have issue, it takes the last client.id config
and report the metrics.
1) Does it really report the Request rate per producer or all the
producers to all the brokers (*AllBrokersProducerRequestRateAndTimeMs or
AllBrokersProducerRequestSize *) Is it like global aggregated Request
rate and size across all topics for All Brokers ?
2) Similarly under "*ProducerTopicMetrics*", AllTopicsBytesPerSec ,
AllTopicsDroppedMessagesPerSec, AllTopicsMessagesPerSec are per topic or
across all topics ? The numbers for AllTopics* are same as per topoic
(logmon.error-*) ? Can you guys please clarify this ? Do you guys
intended to work as global aggregated for ALL topic ?
Please see the following image:
Thanks,
Bhavesh
On Wed, May 28, 2014 at 9:44 PM, Jun Rao <ju...@gmail.com> wrote:
> The producer has a "failedToSend" jmx, which includes all kinds of errors.
> We don't have the message rate per partition.
>
> Thanks,
>
> Jun
>
>
> On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <
> mistry.p.bhavesh@gmail.com
> > wrote:
>
> > Hi Kafka Group,
> >
> > I need to get following metrics from the producer side.
> >
> > I am able to get following metric using the ProducerTopicMetrics class
> per
> > minute.
> > messageRate
> > byteRate
> > droppedMessageRate
> >
> > I would like to know how to get above metric per topic per partition.
> > Also, how do I get count of number of exception occurred due to
> > kafka.common.MessageSizeTooLargeException, or Network
> > Down or message rejected due to Broker being down etc. Is there a JMX
> or
> > any other bean to get this metric counter ?
> >
> > We are using Kafka version 0.8.0
> >
> > Thanks,
> >
> > Bhavesh
> >
>
Re: Producer Side Metric Details
Posted by Jun Rao <ju...@gmail.com>.
The producer has a "failedToSend" jmx, which includes all kinds of errors.
We don't have the message rate per partition.
Thanks,
Jun
On Wed, May 28, 2014 at 11:41 AM, Bhavesh Mistry <mistry.p.bhavesh@gmail.com
> wrote:
> Hi Kafka Group,
>
> I need to get following metrics from the producer side.
>
> I am able to get following metric using the ProducerTopicMetrics class per
> minute.
> messageRate
> byteRate
> droppedMessageRate
>
> I would like to know how to get above metric per topic per partition.
> Also, how do I get count of number of exception occurred due to
> kafka.common.MessageSizeTooLargeException, or Network
> Down or message rejected due to Broker being down etc. Is there a JMX or
> any other bean to get this metric counter ?
>
> We are using Kafka version 0.8.0
>
> Thanks,
>
> Bhavesh
>