You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Akshay Kumar <ak...@ameyo.com.INVALID> on 2023/05/07 19:30:23 UTC
Kafka Consumer Lag Monitoring
Hello team,
I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I wanted to
monitor consumer lag, so I was using Burrow for that, but I am unable to
use Burrow without Zookeeper.
Does Burror work without Zookeeper?
Or what is the better or best way to monitor consumer lag and lag history
for multiple topics, consumers, and consumer groups?
All the topics are consumed by different-different consumers(applications).
[image: https://exotel.com/]
Akshay Kumar
Member of Technical Staff - 2 | Exotel
<https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature>
[image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter]
<https://twitter.com/Exotel> [image: Linkedin]
<https://www.linkedin.com/company/exotel-techcom-private-limited/>
Ph: +918556063696
Latest from the Exotel Blog
<https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/>
--
<https://exotel.com/>
*CONFIDENTIALITY NOTE:* This e-mail is intended only
for the person or entity to which it is addressed and contains information
that is privileged, confidential, or otherwise protected from disclosure.
Dissemination, distribution, or copying of this e-mail or the information
contained herein by anyone other than the intended recipient, or an
employee or agent responsible for delivering the message to the intended
recipient, is prohibited. If you have received this e-mail in error, please
delete this message and immediately notify the sender by e-mail.
*NOTE:*
This e-mail does not constitute an electronic signature and the sender does
not intend to enter into any agreement by way of this e-mail, unless
otherwise expressly provided by the sender within this e-mail.
Re: Kafka Consumer Lag Monitoring
Posted by Akshay Kumar <ak...@ameyo.com.INVALID>.
Hello team,
JMX might not help as Kafka JMX doesn't provide Kafka.consumer Mbeans.
However, this Mbean is provided by consumer application. So I am trying to
fetch JMX data from the consumer application with the help of Zabbix, but
I'm unable to fetch proper data.
JMX keys tried -
-
jmx.discovery[beans,"kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-cg-campaign-customer-campaign-customer-consumer-group*"]
-
jmx.get[beans,"kafka.consumer:type=consumer-fetch-manager-metrics,client-id=consumer-cg-campaign-customer-campaign-customer-consumer-group*"]
Can anyone provide any other tool recommendations (tool should work without
zookeeper) or commands which can provide lag for any historical instance?
[image: https://exotel.com/]
Akshay Kumar
Member of Technical Staff - 2 | Exotel
<https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature>
[image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter]
<https://twitter.com/Exotel> [image: Linkedin]
<https://www.linkedin.com/company/exotel-techcom-private-limited/>
Ph: +918556063696
Latest from the Exotel Blog
<https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/>
On Wed, May 10, 2023 at 12:28 AM Santhosh Kumar <gr...@gmail.com>
wrote:
> Hi Akshay
>
> Burrow is a popular open-source tool for monitoring the consumer lag in
> Apache Kafka. However, Burrow depends on ZooKeeper to access the Kafka
> metadata and track the consumer lag. Therefore, it is not possible to use
> Burrow without ZooKeeper.
>
> Since you are using Kafka version 3.3.1, you can leverage Kafka's built-in
> consumer lag monitoring functionality without the need for any additional
> tools or dependencies. Kafka exposes consumer lag metrics through JMX, and
> you can use any monitoring system that supports JMX to collect and
> visualize these metrics.
>
> Here are the steps to monitor consumer lag in Kafka:
>
> 1.
>
> Enable JMX on Kafka brokers and consumers by setting the following JVM
> system properties:
>
> -Dcom.sun.management.jmxremote=true
> -Dcom.sun.management.jmxremote.authenticate=false
> -Dcom.sun.management.jmxremote.ssl=false
> -Dcom.sun.management.jmxremote.local.only=false
> -Djava.rmi.server.hostname=<hostname>
>
>
> 1.
>
> Replace <hostname> with the hostname or IP address of the Kafka broker
> or consumer machine.
> 2.
>
> Start a monitoring system that supports JMX, such as Prometheus,
> Grafana, or Datadog.
> 3.
>
> Configure the monitoring system to collect Kafka consumer lag metrics
> using JMX.
>
> For example, if you are using Prometheus, you can use the jmx_exporter
> tool to expose the Kafka JMX metrics as Prometheus metrics.
> 4.
>
> Visualize the Kafka consumer lag metrics in the monitoring system
> dashboard.
>
> You can create custom dashboards that show the consumer lag for
> different topics, consumers, and consumer groups. You can also set up
> alerts to notify you when the consumer lag exceeds a certain threshold.
>
> By using Kafka's built-in JMX metrics and a monitoring system that supports
> JMX, you can monitor consumer lag without the need for additional tools or
> dependencies. This approach also provides a more lightweight and scalable
> solution compared to using external tools like Burrow.
> Thanks
> Santhosh Gopal
> Advisor - Architecture,Performance Engineering
>
> On Tue, May 9, 2023 at 2:14 PM Akshay Kumar <akshaykumar@ameyo.com.invalid
> >
> wrote:
>
> > Hello team,
> >
> > I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I wanted
> to
> > monitor consumer lag, so I was using Burrow for that, but I am unable to
> > use Burrow without Zookeeper.
> >
> > Does Burror work without Zookeeper?
> >
> > Or what is the better or best way to monitor consumer lag and lag history
> > for multiple topics, consumers, and consumer groups?
> >
> > All the topics are consumed by different-different
> consumers(applications).
> >
> > [image: https://exotel.com/]
> > Akshay Kumar
> > Member of Technical Staff - 2 | Exotel
> > <
> https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature
> >
> > [image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter]
> > <https://twitter.com/Exotel> [image: Linkedin]
> > <https://www.linkedin.com/company/exotel-techcom-private-limited/>
> > Ph: +918556063696
> > Latest from the Exotel Blog
> > <
> https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/
> >
> >
> > [image: https://exotel.com/] <https://exotel.com/>
> > *CONFIDENTIALITY NOTE:* This e-mail is intended only for the person or
> > entity to which it is addressed and contains information that is
> > privileged, confidential, or otherwise protected from disclosure.
> > Dissemination, distribution, or copying of this e-mail or the information
> > contained herein by anyone other than the intended recipient, or an
> > employee or agent responsible for delivering the message to the intended
> > recipient, is prohibited. If you have received this e-mail in error,
> please
> > delete this message and immediately notify the sender by e-mail.
> > *NOTE:* This e-mail does not constitute an electronic signature and the
> > sender does not intend to enter into any agreement by way of this e-mail,
> > unless otherwise expressly provided by the sender within this e-mail.
> >
>
--
<https://exotel.com/>
*CONFIDENTIALITY NOTE:* This e-mail is intended only
for the person or entity to which it is addressed and contains information
that is privileged, confidential, or otherwise protected from disclosure.
Dissemination, distribution, or copying of this e-mail or the information
contained herein by anyone other than the intended recipient, or an
employee or agent responsible for delivering the message to the intended
recipient, is prohibited. If you have received this e-mail in error, please
delete this message and immediately notify the sender by e-mail.
*NOTE:*
This e-mail does not constitute an electronic signature and the sender does
not intend to enter into any agreement by way of this e-mail, unless
otherwise expressly provided by the sender within this e-mail.
Re: Kafka Consumer Lag Monitoring
Posted by Santhosh Kumar <gr...@gmail.com>.
Hi Akshay
Burrow is a popular open-source tool for monitoring the consumer lag in
Apache Kafka. However, Burrow depends on ZooKeeper to access the Kafka
metadata and track the consumer lag. Therefore, it is not possible to use
Burrow without ZooKeeper.
Since you are using Kafka version 3.3.1, you can leverage Kafka's built-in
consumer lag monitoring functionality without the need for any additional
tools or dependencies. Kafka exposes consumer lag metrics through JMX, and
you can use any monitoring system that supports JMX to collect and
visualize these metrics.
Here are the steps to monitor consumer lag in Kafka:
1.
Enable JMX on Kafka brokers and consumers by setting the following JVM
system properties:
-Dcom.sun.management.jmxremote=true
-Dcom.sun.management.jmxremote.authenticate=false
-Dcom.sun.management.jmxremote.ssl=false
-Dcom.sun.management.jmxremote.local.only=false
-Djava.rmi.server.hostname=<hostname>
1.
Replace <hostname> with the hostname or IP address of the Kafka broker
or consumer machine.
2.
Start a monitoring system that supports JMX, such as Prometheus,
Grafana, or Datadog.
3.
Configure the monitoring system to collect Kafka consumer lag metrics
using JMX.
For example, if you are using Prometheus, you can use the jmx_exporter
tool to expose the Kafka JMX metrics as Prometheus metrics.
4.
Visualize the Kafka consumer lag metrics in the monitoring system
dashboard.
You can create custom dashboards that show the consumer lag for
different topics, consumers, and consumer groups. You can also set up
alerts to notify you when the consumer lag exceeds a certain threshold.
By using Kafka's built-in JMX metrics and a monitoring system that supports
JMX, you can monitor consumer lag without the need for additional tools or
dependencies. This approach also provides a more lightweight and scalable
solution compared to using external tools like Burrow.
Thanks
Santhosh Gopal
Advisor - Architecture,Performance Engineering
On Tue, May 9, 2023 at 2:14 PM Akshay Kumar <ak...@ameyo.com.invalid>
wrote:
> Hello team,
>
> I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I wanted to
> monitor consumer lag, so I was using Burrow for that, but I am unable to
> use Burrow without Zookeeper.
>
> Does Burror work without Zookeeper?
>
> Or what is the better or best way to monitor consumer lag and lag history
> for multiple topics, consumers, and consumer groups?
>
> All the topics are consumed by different-different consumers(applications).
>
> [image: https://exotel.com/]
> Akshay Kumar
> Member of Technical Staff - 2 | Exotel
> <https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature>
> [image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter]
> <https://twitter.com/Exotel> [image: Linkedin]
> <https://www.linkedin.com/company/exotel-techcom-private-limited/>
> Ph: +918556063696
> Latest from the Exotel Blog
> <https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/>
>
> [image: https://exotel.com/] <https://exotel.com/>
> *CONFIDENTIALITY NOTE:* This e-mail is intended only for the person or
> entity to which it is addressed and contains information that is
> privileged, confidential, or otherwise protected from disclosure.
> Dissemination, distribution, or copying of this e-mail or the information
> contained herein by anyone other than the intended recipient, or an
> employee or agent responsible for delivering the message to the intended
> recipient, is prohibited. If you have received this e-mail in error, please
> delete this message and immediately notify the sender by e-mail.
> *NOTE:* This e-mail does not constitute an electronic signature and the
> sender does not intend to enter into any agreement by way of this e-mail,
> unless otherwise expressly provided by the sender within this e-mail.
>
Re: Kafka Consumer Lag Monitoring
Posted by Raphael Mazelier <ra...@futomaki.net>.
I'm not sure to follow you:
either mine
- https://github.com/ut0mt8/yakle or
- https://github.com/danielqsj/kafka_exporter or
- https://github.com/redpanda-data/kminion
export consumer-group lag metrics. All works.
best,
--
Raphael Mazelier
On 10/05/2023 22:47, Akshay Kumar wrote:
> Hello team,
>
> The Kakfa lag Prometheus exporter is also not providing lag-related
> metrics.
> https://exotel.com/
>
> Akshay Kumar
> Member of Technical Staff - 2 | Exotel
> <https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature>
> Facebook <https://www.facebook.com/Exotel/>Twitter
> <https://twitter.com/Exotel>Linkedin
> <https://www.linkedin.com/company/exotel-techcom-private-limited/>
> Ph: +91855606369
> Latest from the Exotel Blog
> <https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/>
>
>
>
>
Re: Kafka Consumer Lag Monitoring
Posted by Akshay Kumar <ak...@ameyo.com.INVALID>.
Hello team,
The Kakfa lag Prometheus exporter is also not providing lag-related metrics.
[image: https://exotel.com/]
Akshay Kumar
Member of Technical Staff - 2 | Exotel
<https://www.exotel.com/?utm_source=global-db&utm_medium=email&utm_campaign=email-signature&utm_term=dg-int&utm_content=shiva-signature>
[image: Facebook] <https://www.facebook.com/Exotel/> [image: Twitter]
<https://twitter.com/Exotel> [image: Linkedin]
<https://www.linkedin.com/company/exotel-techcom-private-limited/>
Ph: +918556063696
Latest from the Exotel Blog
<https://exotel.com/blog/how-does-an-omnichannel-approach-impact-customer-service/>
On Wed, May 10, 2023 at 12:26 AM raph <ra...@futomaki.net> wrote:
> There is a lot of kakfa lag prometheus exporter. You could give a try at
> mine (named yakle working fine in production since years)But others does
> the job as well.Sent from my Galaxy
> -------- Original message --------From: Akshay Kumar
> <ak...@ameyo.com.INVALID> Date: 5/9/23 20:14 (GMT+01:00) To:
> users@kafka.apache.org Subject: Kafka Consumer Lag Monitoring Hello
> team,I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I
> wanted to monitor consumer lag, so I was using Burrow for that, but I am
> unable to use Burrow without Zookeeper.Does Burror work without
> Zookeeper?Or what is the better or best way to monitor consumer lag and lag
> history for multiple topics, consumers, and consumer groups?All the topics
> are consumed by different-different consumers(applications). Akshay
> KumarMember of Technical Staff - 2 | Exotel Ph:
> +918556063696Latest from the Exotel Blog
>
>
> CONFIDENTIALITY NOTE: This e-mail is intended only for the person or
> entity to which it is addressed and contains information that is
> privileged, confidential, or otherwise protected from disclosure.
> Dissemination, distribution, or copying of this e-mail or the information
> contained herein by anyone other than the intended recipient, or an
> employee or agent responsible for delivering the message to the intended
> recipient, is prohibited. If you have received this e-mail in error, please
> delete this message and immediately notify the sender by e-mail. NOTE: This
> e-mail does not constitute an electronic signature and the sender does not
> intend to enter into any agreement by way of this e-mail, unless otherwise
> expressly provided by the sender within this e-mail.
--
<https://exotel.com/>
*CONFIDENTIALITY NOTE:* This e-mail is intended only
for the person or entity to which it is addressed and contains information
that is privileged, confidential, or otherwise protected from disclosure.
Dissemination, distribution, or copying of this e-mail or the information
contained herein by anyone other than the intended recipient, or an
employee or agent responsible for delivering the message to the intended
recipient, is prohibited. If you have received this e-mail in error, please
delete this message and immediately notify the sender by e-mail.
*NOTE:*
This e-mail does not constitute an electronic signature and the sender does
not intend to enter into any agreement by way of this e-mail, unless
otherwise expressly provided by the sender within this e-mail.
RE: Kafka Consumer Lag Monitoring
Posted by raph <ra...@futomaki.net>.
There is a lot of kakfa lag prometheus exporter. You could give a try at mine (named yakle working fine in production since years)But others does the job as well.Sent from my Galaxy
-------- Original message --------From: Akshay Kumar <ak...@ameyo.com.INVALID> Date: 5/9/23 20:14 (GMT+01:00) To: users@kafka.apache.org Subject: Kafka Consumer Lag Monitoring Hello team,I am using Zookeeper less Kafka (Kafka Kraft - version 3.3.1). I wanted to monitor consumer lag, so I was using Burrow for that, but I am unable to use Burrow without Zookeeper.Does Burror work without Zookeeper?Or what is the better or best way to monitor consumer lag and lag history for multiple topics, consumers, and consumer groups?All the topics are consumed by different-different consumers(applications). Akshay KumarMember of Technical Staff - 2 | Exotel Ph: +918556063696Latest from the Exotel Blog
CONFIDENTIALITY NOTE: This e-mail is intended only for the person or entity to which it is addressed and contains information that is privileged, confidential, or otherwise protected from disclosure. Dissemination, distribution, or copying of this e-mail or the information contained herein by anyone other than the intended recipient, or an employee or agent responsible for delivering the message to the intended recipient, is prohibited. If you have received this e-mail in error, please delete this message and immediately notify the sender by e-mail. NOTE: This e-mail does not constitute an electronic signature and the sender does not intend to enter into any agreement by way of this e-mail, unless otherwise expressly provided by the sender within this e-mail.