You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sunil CHAUDHARI <su...@dbs.com.INVALID> on 2020/02/20 10:51:10 UTC
RE: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Hi Liam Clarke,
Sorry but this is bit unclear for me.
Can you please elaborate your answer? I am beginner to Kafka.
" Producers emit metrics via JMX ":
- How to enable this? I have kafka-Manager. Can I make use of kafka-manager? How?
“kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate” please help to explain this.
Regards,
Sunil.
-----Original Message-----
From: Liam Clarke <li...@adscale.co.nz>
Sent: Thursday, February 20, 2020 11:16 AM
To: users@kafka.apache.org
Subject: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
This mail originated from an external party outside DBS - mailto:users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
Hi Sunil,
Producers emit metrics via JMX that will help you, assuming that your producers are using a round robin partition assignment strategy, you could divide this metric by your number of partitions,
kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate
Kind regards,
Liam Clarke
On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <ma...@dbs.com.invalid> wrote:
> Hi
> I was referring to the article by Mr. June Rao about partitions in
> kafka cluster.
> https://www.confluent.io/blog/how-choose-number-topics-partitions-kafk
> a-cluster/
>
> "A rough formula for picking the number of partitions is based on
> throughput. You measure the throughout that you can achieve on a
> single partition for production (call it p) and consumption (call it
> c). Let's say your target throughput is t. Then you need to have at
> least max(t/p, t/c) partitions."
>
> I have the data pipeline as below.
>
> Filebeat-->Kafka-->Logstash-->Elasticsearch
> There are many filebeat agents sending data to kafka. I want to
> understand , how can I measure the events per seconds getting written
> to Kafka? This will help me to know 'p' in above formula.
> I can measure the consumer throughput by monitoring logsatsh pipelines
> on Kibana. So it will give me 'c' in above formula.
>
> I know target throughput in my cluster, that is 't'. 30k events/s.
>
> Please let me know if I am going wrong?
>
> Regards,
> Sunil.
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use
> of the individual or entity named above and may contain information
> that is privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of
> this communication is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and delete the mail. Thank you.
>
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.
Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
Posted by George Leonard <ge...@gmail.com>.
I’m going to say no...
Metric beat expose OS metrics. You talking about a JVM here exposing values via JMX.
Have you looked at catching all of this using Prometheus and dash boarding it using Grafana, as a real time dashboard.
Sent from my iPhone
George Leonard
__________________________
Georgelza@gmail.com
+27 82 655 2466
> On 21 Feb 2020, at 07:16, Sunil CHAUDHARI <su...@dbs.com.invalid> wrote:
>
> Hi Liam Clarke,
> Thanks for this elaboration.
> Surely I will google.
> One more question about Mbeans. If I am capturing system matrix on kafka broker using metricbeat, then is it possible that I will get those Mbeans?
> I know this is out of the topic, but in general, if I capture JVM metric with any of the monitoring tools, then will I get it?
>
>
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Friday, February 21, 2020 1:03 AM
> To: users@kafka.apache.org
> Subject: Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
>
> This mail originated from an external party outside DBS - users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
>
> The metrics are exposed in the JVM the producer is running within as Mbeans. The long string I gave you is the relevant MBean object name. You can connect to the JVM using JConsole to view the MBeans. There are also multiple libraries that will scrape a JVM via JMX to extract values from MBeans.
>
> If you're not familiar with JMX or JConsole, there's plenty of great documentation on the Internet, have a Google :)
>
>> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, <su...@dbs.com.invalid> wrote:
>>
>> Hi Liam Clarke,
>> Sorry but this is bit unclear for me.
>> Can you please elaborate your answer? I am beginner to Kafka.
>> " Producers emit metrics via JMX ":
>> - How to enable this? I have kafka-Manager. Can I make use of
>> kafka-manager? How?
>> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
>> please help to explain this.
>>
>> Regards,
>> Sunil.
>>
>> -----Original Message-----
>> From: Liam Clarke <li...@adscale.co.nz>
>> Sent: Thursday, February 20, 2020 11:16 AM
>> To: users@kafka.apache.org
>> Subject: [External] Re: Urgent helep please! How to measure producer
>> and consumer throughput for single partition?
>>
>> This mail originated from an external party outside DBS - mailto:
>> users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not
>> click on links or open attachments unless you recognize the sender and
>> know the content is safe.
>>
>> Hi Sunil,
>>
>> Producers emit metrics via JMX that will help you, assuming that your
>> producers are using a round robin partition assignment strategy, you
>> could divide this metric by your number of partitions,
>>
>>
>> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s
>> end-rate
>>
>> Kind regards,
>>
>> Liam Clarke
>>
>> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
>> sunilchaudhari@dbs.com.invalid> wrote:
>>
>>> Hi
>>> I was referring to the article by Mr. June Rao about partitions in
>>> kafka cluster.
>>> https://www.confluent.io/blog/how-choose-number-topics-partitions-ka
>>> fk
>>> a-cluster/
>>>
>>> "A rough formula for picking the number of partitions is based on
>>> throughput. You measure the throughout that you can achieve on a
>>> single partition for production (call it p) and consumption (call it
>>> c). Let's say your target throughput is t. Then you need to have at
>>> least max(t/p, t/c) partitions."
>>>
>>> I have the data pipeline as below.
>>>
>>> Filebeat-->Kafka-->Logstash-->Elasticsearch
>>> There are many filebeat agents sending data to kafka. I want to
>>> understand , how can I measure the events per seconds getting
>>> written to Kafka? This will help me to know 'p' in above formula.
>>> I can measure the consumer throughput by monitoring logsatsh
>>> pipelines on Kibana. So it will give me 'c' in above formula.
>>>
>>> I know target throughput in my cluster, that is 't'. 30k events/s.
>>>
>>> Please let me know if I am going wrong?
>>>
>>> Regards,
>>> Sunil.
>>> CONFIDENTIAL NOTE:
>>> The information contained in this email is intended only for the use
>>> of the individual or entity named above and may contain information
>>> that is privileged, confidential and exempt from disclosure under
>> applicable law.
>>> If the reader of this message is not the intended recipient, you are
>>> hereby notified that any dissemination, distribution or copying of
>>> this communication is strictly prohibited. If you have received this
>>> message in error, please immediately notify the sender and delete
>>> the
>> mail. Thank you.
>>>
>>
>>
>> CONFIDENTIAL NOTE:
>> The information contained in this email is intended only for the use
>> of the individual or entity named above and may contain information
>> that is privileged, confidential and exempt from disclosure under applicable law.
>> If the reader of this message is not the intended recipient, you are
>> hereby notified that any dissemination, distribution or copying of
>> this communication is strictly prohibited. If you have received this
>> message in error, please immediately notify the sender and delete the mail. Thank you.
>>
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.
Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
Posted by George Leonard <ge...@gmail.com>.
I stand corrected...
Sent from my iPhone
George Leonard
__________________________
Georgelza@gmail.com
+27 82 655 2466
> On 21 Feb 2020, at 07:22, Liam Clarke <li...@adscale.co.nz> wrote:
>
> Hi Sunil,
>
> Looks like Metricbeats has a Jolokia module that will capture JMX exposed
> metrics for you:
> https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx
>
> Kind regards,
>
> Liam Clarke
>
>> On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI
>> <su...@dbs.com.invalid> wrote:
>>
>> Hi Liam Clarke,
>> Thanks for this elaboration.
>> Surely I will google.
>> One more question about Mbeans. If I am capturing system matrix on kafka
>> broker using metricbeat, then is it possible that I will get those Mbeans?
>> I know this is out of the topic, but in general, if I capture JVM metric
>> with any of the monitoring tools, then will I get it?
>>
>>
>>
>> -----Original Message-----
>> From: Liam Clarke <li...@adscale.co.nz>
>> Sent: Friday, February 21, 2020 1:03 AM
>> To: users@kafka.apache.org
>> Subject: Re: [External] Re: Urgent helep please! How to measure producer
>> and consumer throughput for single partition?
>>
>> This mail originated from an external party outside DBS -
>> users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org. Do not click
>> on links or open attachments unless you recognize the sender and know the
>> content is safe.
>>
>> The metrics are exposed in the JVM the producer is running within as
>> Mbeans. The long string I gave you is the relevant MBean object name. You
>> can connect to the JVM using JConsole to view the MBeans. There are also
>> multiple libraries that will scrape a JVM via JMX to extract values from
>> MBeans.
>>
>> If you're not familiar with JMX or JConsole, there's plenty of great
>> documentation on the Internet, have a Google :)
>>
>> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, <su...@dbs.com.invalid>
>> wrote:
>>
>>> Hi Liam Clarke,
>>> Sorry but this is bit unclear for me.
>>> Can you please elaborate your answer? I am beginner to Kafka.
>>> " Producers emit metrics via JMX ":
>>> - How to enable this? I have kafka-Manager. Can I make use of
>>> kafka-manager? How?
>>>
>> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
>>> please help to explain this.
>>>
>>> Regards,
>>> Sunil.
>>>
>>> -----Original Message-----
>>> From: Liam Clarke <li...@adscale.co.nz>
>>> Sent: Thursday, February 20, 2020 11:16 AM
>>> To: users@kafka.apache.org
>>> Subject: [External] Re: Urgent helep please! How to measure producer
>>> and consumer throughput for single partition?
>>>
>>> This mail originated from an external party outside DBS - mailto:
>>> users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not
>>> click on links or open attachments unless you recognize the sender and
>>> know the content is safe.
>>>
>>> Hi Sunil,
>>>
>>> Producers emit metrics via JMX that will help you, assuming that your
>>> producers are using a round robin partition assignment strategy, you
>>> could divide this metric by your number of partitions,
>>>
>>>
>>> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s
>>> end-rate
>>>
>>> Kind regards,
>>>
>>> Liam Clarke
>>>
>>> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
>>> sunilchaudhari@dbs.com.invalid> wrote:
>>>
>>>> Hi
>>>> I was referring to the article by Mr. June Rao about partitions in
>>>> kafka cluster.
>>>> https://www.confluent.io/blog/how-choose-number-topics-partitions-ka
>>>> fk
>>>> a-cluster/
>>>>
>>>> "A rough formula for picking the number of partitions is based on
>>>> throughput. You measure the throughout that you can achieve on a
>>>> single partition for production (call it p) and consumption (call it
>>>> c). Let's say your target throughput is t. Then you need to have at
>>>> least max(t/p, t/c) partitions."
>>>>
>>>> I have the data pipeline as below.
>>>>
>>>> Filebeat-->Kafka-->Logstash-->Elasticsearch
>>>> There are many filebeat agents sending data to kafka. I want to
>>>> understand , how can I measure the events per seconds getting
>>>> written to Kafka? This will help me to know 'p' in above formula.
>>>> I can measure the consumer throughput by monitoring logsatsh
>>>> pipelines on Kibana. So it will give me 'c' in above formula.
>>>>
>>>> I know target throughput in my cluster, that is 't'. 30k events/s.
>>>>
>>>> Please let me know if I am going wrong?
>>>>
>>>> Regards,
>>>> Sunil.
>>>> CONFIDENTIAL NOTE:
>>>> The information contained in this email is intended only for the use
>>>> of the individual or entity named above and may contain information
>>>> that is privileged, confidential and exempt from disclosure under
>>> applicable law.
>>>> If the reader of this message is not the intended recipient, you are
>>>> hereby notified that any dissemination, distribution or copying of
>>>> this communication is strictly prohibited. If you have received this
>>>> message in error, please immediately notify the sender and delete
>>>> the
>>> mail. Thank you.
>>>>
>>>
>>>
>>> CONFIDENTIAL NOTE:
>>> The information contained in this email is intended only for the use
>>> of the individual or entity named above and may contain information
>>> that is privileged, confidential and exempt from disclosure under
>> applicable law.
>>> If the reader of this message is not the intended recipient, you are
>>> hereby notified that any dissemination, distribution or copying of
>>> this communication is strictly prohibited. If you have received this
>>> message in error, please immediately notify the sender and delete the
>> mail. Thank you.
>>>
>>
>> CONFIDENTIAL NOTE:
>> The information contained in this email is intended only for the use of
>> the individual or entity named above and may contain information that is
>> privileged, confidential and exempt from disclosure under applicable law.
>> If the reader of this message is not the intended recipient, you are hereby
>> notified that any dissemination, distribution or copying of this
>> communication is strictly prohibited. If you have received this message in
>> error, please immediately notify the sender and delete the mail. Thank you.
>>
Re: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Posted by Liam Clarke <li...@adscale.co.nz>.
Hi Sunil,
You'll have to write the object name in a manner that Metricbeats
understand. I'd suggest using JConsole to explore a running app on the JVM
that has JMX enabled and a producer in it, to get an understanding of the
object names you need to use in your Metricbeat conf.
Cheers,
Liam Clarke
On Fri, 21 Feb. 2020, 7:05 pm Sunil CHAUDHARI,
<su...@dbs.com.invalid> wrote:
> Hi Liam and George,
>
> Thanks for support.
>
> Just now I got the same Jolokio from elastic documentation.
>
> Checking its configurations. There are no metrics mentioned by you .
> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate
>
> It seems I have to add few metrics in the *jolokia.yml. *is that
> correct?
>
>
>
>
>
>
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Friday, February 21, 2020 10:52 AM
> To: users@kafka.apache.org
> Subject: Re: [External] Re: Urgent helep please! How to measure producer
> and consumer throughput for single partition?
>
>
>
> This mail originated from an external party outside DBS -
> users-return-39580-sunilchaudhari=dbs.com@kafka.apache.org. Do not click
> on links or open attachments unless you recognize the sender and know the
> content is safe.
>
>
>
> Hi Sunil,
>
>
>
> Looks like Metricbeats has a Jolokia module that will capture JMX exposed
> metrics for you:
>
> https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx
>
>
>
> Kind regards,
>
>
>
> Liam Clarke
>
>
>
> On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI <
> sunilchaudhari@dbs.com.invalid> wrote:
>
>
>
> > Hi Liam Clarke,
>
> > Thanks for this elaboration.
>
> > Surely I will google.
>
> > One more question about Mbeans. If I am capturing system matrix on
>
> > kafka broker using metricbeat, then is it possible that I will get those
> Mbeans?
>
> > I know this is out of the topic, but in general, if I capture JVM
>
> > metric with any of the monitoring tools, then will I get it?
>
> >
>
> >
>
> >
>
> > -----Original Message-----
>
> > From: Liam Clarke <li...@adscale.co.nz>
>
> > Sent: Friday, February 21, 2020 1:03 AM
>
> > To: users@kafka.apache.org
>
> > Subject: Re: [External] Re: Urgent helep please! How to measure
>
> > producer and consumer throughput for single partition?
>
> >
>
> > This mail originated from an external party outside DBS -
>
> > users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org. Do not
>
> > click on links or open attachments unless you recognize the sender and
>
> > know the content is safe.
>
> >
>
> > The metrics are exposed in the JVM the producer is running within as
>
> > Mbeans. The long string I gave you is the relevant MBean object name.
>
> > You can connect to the JVM using JConsole to view the MBeans. There
>
> > are also multiple libraries that will scrape a JVM via JMX to extract
>
> > values from MBeans.
>
> >
>
> > If you're not familiar with JMX or JConsole, there's plenty of great
>
> > documentation on the Internet, have a Google :)
>
> >
>
> > On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI,
>
> > <su...@dbs.com.invalid>
>
> > wrote:
>
> >
>
> > > Hi Liam Clarke,
>
> > > Sorry but this is bit unclear for me.
>
> > > Can you please elaborate your answer? I am beginner to Kafka.
>
> > > " Producers emit metrics via JMX ":
>
> > > - How to enable this? I have kafka-Manager. Can I make use
>
> > > of kafka-manager? How?
>
> > >
>
> >
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
>
> > > please help to explain this.
>
> > >
>
> > > Regards,
>
> > > Sunil.
>
> > >
>
> > > -----Original Message-----
>
> > > From: Liam Clarke <li...@adscale.co.nz>
>
> > > Sent: Thursday, February 20, 2020 11:16 AM
>
> > > To: users@kafka.apache.org
>
> > > Subject: [External] Re: Urgent helep please! How to measure producer
>
> > > and consumer throughput for single partition?
>
> > >
>
> > > This mail originated from an external party outside DBS - mailto:
>
> > > users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not
>
> > > click on links or open attachments unless you recognize the sender
>
> > > and know the content is safe.
>
> > >
>
> > > Hi Sunil,
>
> > >
>
> > > Producers emit metrics via JMX that will help you, assuming that
>
> > > your producers are using a round robin partition assignment
>
> > > strategy, you could divide this metric by your number of partitions,
>
> > >
>
> > >
>
> > > kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record
>
> > > -s
>
> > > end-rate
>
> > >
>
> > > Kind regards,
>
> > >
>
> > > Liam Clarke
>
> > >
>
> > > On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
>
> > > sunilchaudhari@dbs.com.invalid> wrote:
>
> > >
>
> > > > Hi
>
> > > > I was referring to the article by Mr. June Rao about partitions in
>
> > > > kafka cluster.
>
> > > > https://www.confluent.io/blog/how-choose-number-topics-partitions-
>
> > > > ka
>
> > > > fk
>
> > > > a-cluster/
>
> > > >
>
> > > > "A rough formula for picking the number of partitions is based on
>
> > > > throughput. You measure the throughout that you can achieve on a
>
> > > > single partition for production (call it p) and consumption (call
>
> > > > it c). Let's say your target throughput is t. Then you need to
>
> > > > have at least max(t/p, t/c) partitions."
>
> > > >
>
> > > > I have the data pipeline as below.
>
> > > >
>
> > > > Filebeat-->Kafka-->Logstash-->Elasticsearch
>
> > > > There are many filebeat agents sending data to kafka. I want to
>
> > > > understand , how can I measure the events per seconds getting
>
> > > > written to Kafka? This will help me to know 'p' in above formula.
>
> > > > I can measure the consumer throughput by monitoring logsatsh
>
> > > > pipelines on Kibana. So it will give me 'c' in above formula.
>
> > > >
>
> > > > I know target throughput in my cluster, that is 't'. 30k events/s.
>
> > > >
>
> > > > Please let me know if I am going wrong?
>
> > > >
>
> > > > Regards,
>
> > > > Sunil.
>
> > > > CONFIDENTIAL NOTE:
>
> > > > The information contained in this email is intended only for the
>
> > > > use of the individual or entity named above and may contain
>
> > > > information that is privileged, confidential and exempt from
>
> > > > disclosure under
>
> > > applicable law.
>
> > > > If the reader of this message is not the intended recipient, you
>
> > > > are hereby notified that any dissemination, distribution or
>
> > > > copying of this communication is strictly prohibited. If you have
>
> > > > received this message in error, please immediately notify the
>
> > > > sender and delete the
>
> > > mail. Thank you.
>
> > > >
>
> > >
>
> > >
>
> > > CONFIDENTIAL NOTE:
>
> > > The information contained in this email is intended only for the use
>
> > > of the individual or entity named above and may contain information
>
> > > that is privileged, confidential and exempt from disclosure under
>
> > applicable law.
>
> > > If the reader of this message is not the intended recipient, you are
>
> > > hereby notified that any dissemination, distribution or copying of
>
> > > this communication is strictly prohibited. If you have received this
>
> > > message in error, please immediately notify the sender and delete
>
> > > the
>
> > mail. Thank you.
>
> > >
>
> >
>
> > CONFIDENTIAL NOTE:
>
> > The information contained in this email is intended only for the use
>
> > of the individual or entity named above and may contain information
>
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
>
> > If the reader of this message is not the intended recipient, you are
>
> > hereby notified that any dissemination, distribution or copying of
>
> > this communication is strictly prohibited. If you have received this
>
> > message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
> >
>
>
> CONFIDENTIAL NOTE: The information contained in this email is intended
> only for the use of the individual or entity named above and may contain
> information that is privileged, confidential and exempt from disclosure
> under applicable law. If the reader of this message is not the intended
> recipient, you are hereby notified that any dissemination, distribution or
> copying of this communication is strictly prohibited. If you have received
> this message in error, please immediately notify the sender and delete the
> mail. Thank you.
>
RE: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Posted by Sunil CHAUDHARI <su...@dbs.com.INVALID>.
Hi Liam and George,
Thanks for support.
Just now I got the same Jolokio from elastic documentation.
Checking its configurations. There are no metrics mentioned by you . kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate
It seems I have to add few metrics in the jolokia.yml. is that correct?
[cid:image001.png@01D5E8A9.3858D300]
-----Original Message-----
From: Liam Clarke <li...@adscale.co.nz>
Sent: Friday, February 21, 2020 10:52 AM
To: users@kafka.apache.org
Subject: Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
This mail originated from an external party outside DBS - users-return-39580-sunilchaudhari=dbs.com@kafka.apache.org<ma...@kafka.apache.org>. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
Hi Sunil,
Looks like Metricbeats has a Jolokia module that will capture JMX exposed metrics for you:
https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx
Kind regards,
Liam Clarke
On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI <su...@dbs.com.invalid>> wrote:
> Hi Liam Clarke,
> Thanks for this elaboration.
> Surely I will google.
> One more question about Mbeans. If I am capturing system matrix on
> kafka broker using metricbeat, then is it possible that I will get those Mbeans?
> I know this is out of the topic, but in general, if I capture JVM
> metric with any of the monitoring tools, then will I get it?
>
>
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>>
> Sent: Friday, February 21, 2020 1:03 AM
> To: users@kafka.apache.org<ma...@kafka.apache.org>
> Subject: Re: [External] Re: Urgent helep please! How to measure
> producer and consumer throughput for single partition?
>
> This mail originated from an external party outside DBS -
> users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org<ma...@kafka.apache.org>. Do not
> click on links or open attachments unless you recognize the sender and
> know the content is safe.
>
> The metrics are exposed in the JVM the producer is running within as
> Mbeans. The long string I gave you is the relevant MBean object name.
> You can connect to the JVM using JConsole to view the MBeans. There
> are also multiple libraries that will scrape a JVM via JMX to extract
> values from MBeans.
>
> If you're not familiar with JMX or JConsole, there's plenty of great
> documentation on the Internet, have a Google :)
>
> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI,
> <su...@dbs.com.invalid>>
> wrote:
>
> > Hi Liam Clarke,
> > Sorry but this is bit unclear for me.
> > Can you please elaborate your answer? I am beginner to Kafka.
> > " Producers emit metrics via JMX ":
> > - How to enable this? I have kafka-Manager. Can I make use
> > of kafka-manager? How?
> >
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
> > please help to explain this.
> >
> > Regards,
> > Sunil.
> >
> > -----Original Message-----
> > From: Liam Clarke <li...@adscale.co.nz>>
> > Sent: Thursday, February 20, 2020 11:16 AM
> > To: users@kafka.apache.org<ma...@kafka.apache.org>
> > Subject: [External] Re: Urgent helep please! How to measure producer
> > and consumer throughput for single partition?
> >
> > This mail originated from an external party outside DBS - mailto:
> > users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org<ma...@kafka.apache.org>. Do not
> > click on links or open attachments unless you recognize the sender
> > and know the content is safe.
> >
> > Hi Sunil,
> >
> > Producers emit metrics via JMX that will help you, assuming that
> > your producers are using a round robin partition assignment
> > strategy, you could divide this metric by your number of partitions,
> >
> >
> > kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record
> > -s
> > end-rate
> >
> > Kind regards,
> >
> > Liam Clarke
> >
> > On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
> > sunilchaudhari@dbs.com.invalid<ma...@dbs.com.invalid>> wrote:
> >
> > > Hi
> > > I was referring to the article by Mr. June Rao about partitions in
> > > kafka cluster.
> > > https://www.confluent.io/blog/how-choose-number-topics-partitions-
> > > ka
> > > fk
> > > a-cluster/
> > >
> > > "A rough formula for picking the number of partitions is based on
> > > throughput. You measure the throughout that you can achieve on a
> > > single partition for production (call it p) and consumption (call
> > > it c). Let's say your target throughput is t. Then you need to
> > > have at least max(t/p, t/c) partitions."
> > >
> > > I have the data pipeline as below.
> > >
> > > Filebeat-->Kafka-->Logstash-->Elasticsearch
> > > There are many filebeat agents sending data to kafka. I want to
> > > understand , how can I measure the events per seconds getting
> > > written to Kafka? This will help me to know 'p' in above formula.
> > > I can measure the consumer throughput by monitoring logsatsh
> > > pipelines on Kibana. So it will give me 'c' in above formula.
> > >
> > > I know target throughput in my cluster, that is 't'. 30k events/s.
> > >
> > > Please let me know if I am going wrong?
> > >
> > > Regards,
> > > Sunil.
> > > CONFIDENTIAL NOTE:
> > > The information contained in this email is intended only for the
> > > use of the individual or entity named above and may contain
> > > information that is privileged, confidential and exempt from
> > > disclosure under
> > applicable law.
> > > If the reader of this message is not the intended recipient, you
> > > are hereby notified that any dissemination, distribution or
> > > copying of this communication is strictly prohibited. If you have
> > > received this message in error, please immediately notify the
> > > sender and delete the
> > mail. Thank you.
> > >
> >
> >
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use
> > of the individual or entity named above and may contain information
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
> > If the reader of this message is not the intended recipient, you are
> > hereby notified that any dissemination, distribution or copying of
> > this communication is strictly prohibited. If you have received this
> > message in error, please immediately notify the sender and delete
> > the
> mail. Thank you.
> >
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use
> of the individual or entity named above and may contain information
> that is privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of
> this communication is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and delete the mail. Thank you.
>
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.
Re: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Posted by Liam Clarke <li...@adscale.co.nz>.
Hi Sunil,
Looks like Metricbeats has a Jolokia module that will capture JMX exposed
metrics for you:
https://www.elastic.co/blog/brewing-in-beats-add-support-for-jolokia-lmx
Kind regards,
Liam Clarke
On Fri, Feb 21, 2020 at 6:16 PM Sunil CHAUDHARI
<su...@dbs.com.invalid> wrote:
> Hi Liam Clarke,
> Thanks for this elaboration.
> Surely I will google.
> One more question about Mbeans. If I am capturing system matrix on kafka
> broker using metricbeat, then is it possible that I will get those Mbeans?
> I know this is out of the topic, but in general, if I capture JVM metric
> with any of the monitoring tools, then will I get it?
>
>
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Friday, February 21, 2020 1:03 AM
> To: users@kafka.apache.org
> Subject: Re: [External] Re: Urgent helep please! How to measure producer
> and consumer throughput for single partition?
>
> This mail originated from an external party outside DBS -
> users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org. Do not click
> on links or open attachments unless you recognize the sender and know the
> content is safe.
>
> The metrics are exposed in the JVM the producer is running within as
> Mbeans. The long string I gave you is the relevant MBean object name. You
> can connect to the JVM using JConsole to view the MBeans. There are also
> multiple libraries that will scrape a JVM via JMX to extract values from
> MBeans.
>
> If you're not familiar with JMX or JConsole, there's plenty of great
> documentation on the Internet, have a Google :)
>
> On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, <su...@dbs.com.invalid>
> wrote:
>
> > Hi Liam Clarke,
> > Sorry but this is bit unclear for me.
> > Can you please elaborate your answer? I am beginner to Kafka.
> > " Producers emit metrics via JMX ":
> > - How to enable this? I have kafka-Manager. Can I make use of
> > kafka-manager? How?
> >
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
> > please help to explain this.
> >
> > Regards,
> > Sunil.
> >
> > -----Original Message-----
> > From: Liam Clarke <li...@adscale.co.nz>
> > Sent: Thursday, February 20, 2020 11:16 AM
> > To: users@kafka.apache.org
> > Subject: [External] Re: Urgent helep please! How to measure producer
> > and consumer throughput for single partition?
> >
> > This mail originated from an external party outside DBS - mailto:
> > users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not
> > click on links or open attachments unless you recognize the sender and
> > know the content is safe.
> >
> > Hi Sunil,
> >
> > Producers emit metrics via JMX that will help you, assuming that your
> > producers are using a round robin partition assignment strategy, you
> > could divide this metric by your number of partitions,
> >
> >
> > kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s
> > end-rate
> >
> > Kind regards,
> >
> > Liam Clarke
> >
> > On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
> > sunilchaudhari@dbs.com.invalid> wrote:
> >
> > > Hi
> > > I was referring to the article by Mr. June Rao about partitions in
> > > kafka cluster.
> > > https://www.confluent.io/blog/how-choose-number-topics-partitions-ka
> > > fk
> > > a-cluster/
> > >
> > > "A rough formula for picking the number of partitions is based on
> > > throughput. You measure the throughout that you can achieve on a
> > > single partition for production (call it p) and consumption (call it
> > > c). Let's say your target throughput is t. Then you need to have at
> > > least max(t/p, t/c) partitions."
> > >
> > > I have the data pipeline as below.
> > >
> > > Filebeat-->Kafka-->Logstash-->Elasticsearch
> > > There are many filebeat agents sending data to kafka. I want to
> > > understand , how can I measure the events per seconds getting
> > > written to Kafka? This will help me to know 'p' in above formula.
> > > I can measure the consumer throughput by monitoring logsatsh
> > > pipelines on Kibana. So it will give me 'c' in above formula.
> > >
> > > I know target throughput in my cluster, that is 't'. 30k events/s.
> > >
> > > Please let me know if I am going wrong?
> > >
> > > Regards,
> > > Sunil.
> > > CONFIDENTIAL NOTE:
> > > The information contained in this email is intended only for the use
> > > of the individual or entity named above and may contain information
> > > that is privileged, confidential and exempt from disclosure under
> > applicable law.
> > > If the reader of this message is not the intended recipient, you are
> > > hereby notified that any dissemination, distribution or copying of
> > > this communication is strictly prohibited. If you have received this
> > > message in error, please immediately notify the sender and delete
> > > the
> > mail. Thank you.
> > >
> >
> >
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use
> > of the individual or entity named above and may contain information
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
> > If the reader of this message is not the intended recipient, you are
> > hereby notified that any dissemination, distribution or copying of
> > this communication is strictly prohibited. If you have received this
> > message in error, please immediately notify the sender and delete the
> mail. Thank you.
> >
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use of
> the individual or entity named above and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this message in
> error, please immediately notify the sender and delete the mail. Thank you.
>
RE: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Posted by Sunil CHAUDHARI <su...@dbs.com.INVALID>.
Hi Liam Clarke,
Thanks for this elaboration.
Surely I will google.
One more question about Mbeans. If I am capturing system matrix on kafka broker using metricbeat, then is it possible that I will get those Mbeans?
I know this is out of the topic, but in general, if I capture JVM metric with any of the monitoring tools, then will I get it?
-----Original Message-----
From: Liam Clarke <li...@adscale.co.nz>
Sent: Friday, February 21, 2020 1:03 AM
To: users@kafka.apache.org
Subject: Re: [External] Re: Urgent helep please! How to measure producer and consumer throughput for single partition?
This mail originated from an external party outside DBS - users-return-39570-sunilchaudhari=dbs.com@kafka.apache.org. Do not click on links or open attachments unless you recognize the sender and know the content is safe.
The metrics are exposed in the JVM the producer is running within as Mbeans. The long string I gave you is the relevant MBean object name. You can connect to the JVM using JConsole to view the MBeans. There are also multiple libraries that will scrape a JVM via JMX to extract values from MBeans.
If you're not familiar with JMX or JConsole, there's plenty of great documentation on the Internet, have a Google :)
On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI, <su...@dbs.com.invalid> wrote:
> Hi Liam Clarke,
> Sorry but this is bit unclear for me.
> Can you please elaborate your answer? I am beginner to Kafka.
> " Producers emit metrics via JMX ":
> - How to enable this? I have kafka-Manager. Can I make use of
> kafka-manager? How?
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
> please help to explain this.
>
> Regards,
> Sunil.
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Thursday, February 20, 2020 11:16 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Urgent helep please! How to measure producer
> and consumer throughput for single partition?
>
> This mail originated from an external party outside DBS - mailto:
> users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not
> click on links or open attachments unless you recognize the sender and
> know the content is safe.
>
> Hi Sunil,
>
> Producers emit metrics via JMX that will help you, assuming that your
> producers are using a round robin partition assignment strategy, you
> could divide this metric by your number of partitions,
>
>
> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-s
> end-rate
>
> Kind regards,
>
> Liam Clarke
>
> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
> sunilchaudhari@dbs.com.invalid> wrote:
>
> > Hi
> > I was referring to the article by Mr. June Rao about partitions in
> > kafka cluster.
> > https://www.confluent.io/blog/how-choose-number-topics-partitions-ka
> > fk
> > a-cluster/
> >
> > "A rough formula for picking the number of partitions is based on
> > throughput. You measure the throughout that you can achieve on a
> > single partition for production (call it p) and consumption (call it
> > c). Let's say your target throughput is t. Then you need to have at
> > least max(t/p, t/c) partitions."
> >
> > I have the data pipeline as below.
> >
> > Filebeat-->Kafka-->Logstash-->Elasticsearch
> > There are many filebeat agents sending data to kafka. I want to
> > understand , how can I measure the events per seconds getting
> > written to Kafka? This will help me to know 'p' in above formula.
> > I can measure the consumer throughput by monitoring logsatsh
> > pipelines on Kibana. So it will give me 'c' in above formula.
> >
> > I know target throughput in my cluster, that is 't'. 30k events/s.
> >
> > Please let me know if I am going wrong?
> >
> > Regards,
> > Sunil.
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use
> > of the individual or entity named above and may contain information
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
> > If the reader of this message is not the intended recipient, you are
> > hereby notified that any dissemination, distribution or copying of
> > this communication is strictly prohibited. If you have received this
> > message in error, please immediately notify the sender and delete
> > the
> mail. Thank you.
> >
>
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use
> of the individual or entity named above and may contain information
> that is privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are
> hereby notified that any dissemination, distribution or copying of
> this communication is strictly prohibited. If you have received this
> message in error, please immediately notify the sender and delete the mail. Thank you.
>
CONFIDENTIAL NOTE:
The information contained in this email is intended only for the use of the individual or entity named above and may contain information that is privileged, confidential and exempt from disclosure under applicable law. If the reader of this message is not the intended recipient, you are hereby notified that any dissemination, distribution or copying of this communication is strictly prohibited. If you have received this message in error, please immediately notify the sender and delete the mail. Thank you.
Re: [External] Re: Urgent helep please! How to measure producer and
consumer throughput for single partition?
Posted by Liam Clarke <li...@adscale.co.nz>.
The metrics are exposed in the JVM the producer is running within as
Mbeans. The long string I gave you is the relevant MBean object name. You
can connect to the JVM using JConsole to view the MBeans. There are also
multiple libraries that will scrape a JVM via JMX to extract values from
MBeans.
If you're not familiar with JMX or JConsole, there's plenty of great
documentation on the Internet, have a Google :)
On Thu, 20 Feb. 2020, 11:51 pm Sunil CHAUDHARI,
<su...@dbs.com.invalid> wrote:
> Hi Liam Clarke,
> Sorry but this is bit unclear for me.
> Can you please elaborate your answer? I am beginner to Kafka.
> " Producers emit metrics via JMX ":
> - How to enable this? I have kafka-Manager. Can I make use of
> kafka-manager? How?
> “kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate”
> please help to explain this.
>
> Regards,
> Sunil.
>
> -----Original Message-----
> From: Liam Clarke <li...@adscale.co.nz>
> Sent: Thursday, February 20, 2020 11:16 AM
> To: users@kafka.apache.org
> Subject: [External] Re: Urgent helep please! How to measure producer and
> consumer throughput for single partition?
>
> This mail originated from an external party outside DBS - mailto:
> users-return-39554-sunilchaudhari=dbs.com@kafka.apache.org. Do not click
> on links or open attachments unless you recognize the sender and know the
> content is safe.
>
> Hi Sunil,
>
> Producers emit metrics via JMX that will help you, assuming that your
> producers are using a round robin partition assignment strategy, you could
> divide this metric by your number of partitions,
>
>
> kafka.producer:type=producer-metrics,client-id=(.+),topic=(.+)record-send-rate
>
> Kind regards,
>
> Liam Clarke
>
> On Thu, 20 Feb. 2020, 5:57 pm Sunil CHAUDHARI, <mailto:
> sunilchaudhari@dbs.com.invalid> wrote:
>
> > Hi
> > I was referring to the article by Mr. June Rao about partitions in
> > kafka cluster.
> > https://www.confluent.io/blog/how-choose-number-topics-partitions-kafk
> > a-cluster/
> >
> > "A rough formula for picking the number of partitions is based on
> > throughput. You measure the throughout that you can achieve on a
> > single partition for production (call it p) and consumption (call it
> > c). Let's say your target throughput is t. Then you need to have at
> > least max(t/p, t/c) partitions."
> >
> > I have the data pipeline as below.
> >
> > Filebeat-->Kafka-->Logstash-->Elasticsearch
> > There are many filebeat agents sending data to kafka. I want to
> > understand , how can I measure the events per seconds getting written
> > to Kafka? This will help me to know 'p' in above formula.
> > I can measure the consumer throughput by monitoring logsatsh pipelines
> > on Kibana. So it will give me 'c' in above formula.
> >
> > I know target throughput in my cluster, that is 't'. 30k events/s.
> >
> > Please let me know if I am going wrong?
> >
> > Regards,
> > Sunil.
> > CONFIDENTIAL NOTE:
> > The information contained in this email is intended only for the use
> > of the individual or entity named above and may contain information
> > that is privileged, confidential and exempt from disclosure under
> applicable law.
> > If the reader of this message is not the intended recipient, you are
> > hereby notified that any dissemination, distribution or copying of
> > this communication is strictly prohibited. If you have received this
> > message in error, please immediately notify the sender and delete the
> mail. Thank you.
> >
>
>
> CONFIDENTIAL NOTE:
> The information contained in this email is intended only for the use of
> the individual or entity named above and may contain information that is
> privileged, confidential and exempt from disclosure under applicable law.
> If the reader of this message is not the intended recipient, you are hereby
> notified that any dissemination, distribution or copying of this
> communication is strictly prohibited. If you have received this message in
> error, please immediately notify the sender and delete the mail. Thank you.
>