You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Kafka Life <li...@gmail.com> on 2022/08/15 18:06:05 UTC
Consumer Lag-Apache_kafka_JMX metrics
Dear Kafka Experts
we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
versions of kafka in Grafana.
1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
kafka cluster.
2/ I had thought it is FetcherLag but it looks like it is not as per the
link below.
https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
.
Could one of you experts please guide on which JMX i should use for
consumer lag apart from kafka burrow or such intermediate tools
Thanking you in advance
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by Richard Bosch <ri...@axual.com>.
Hi,
I don't think that the Kafka Broker exposes the current lag as a metric.
It's the reason most of those intermediate tools, like Kafka Exporter,
exist.
The lag information is available to the client itself, but that is based on
the received topic metadata and can be out of date.
Kind regards,
Richard Bosch
Developer Advocate
Axual BV
https://axual.com/
On Mon, Aug 15, 2022 at 8:07 PM Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag but it looks like it is not as per the
> link below.
>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by Kafka Life <li...@gmail.com>.
Thank you Sunil ,Peter Raph and Richard for your kind inputs.Much
appreciated.
On Wed, Aug 17, 2022 at 6:46 AM sunil chaudhari <su...@gmail.com>
wrote:
> You can try this, if you know what prometheus and how its installed
> configured.
>
>
> https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/
>
>
> On Wed, 17 Aug 2022 at 2:25 AM, Peter Bukowinski <pm...@gmail.com> wrote:
>
> > Richard recently answered your query. A kafka cluster does not keep track
> > of lag on behalf of external consumers and it therefore is not available
> in
> > JMX. This is why tools like Burrow were written. The java kafka consumer
> > published consumer lag metrics, and perhaps some other third-party
> clients
> > do, as well.
> >
> > > On Aug 16, 2022, at 12:05 PM, Kafka Life <li...@gmail.com>
> wrote:
> > >
> > > Hello Experts, Any info or pointers on my query please.
> > >
> > >
> > >
> > > On Mon, Aug 15, 2022 at 11:36 PM Kafka Life <li...@gmail.com>
> > wrote:
> > >
> > >> Dear Kafka Experts
> > >> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> > >> versions of kafka in Grafana.
> > >>
> > >> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag
> in
> > >> kafka cluster.
> > >>
> > >> 2/ I had thought it is FetcherLag but it looks like it is not as per
> > the
> > >> link below.
> > >>
> > >>
> >
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> > >> .
> > >>
> > >> Could one of you experts please guide on which JMX i should use for
> > >> consumer lag apart from kafka burrow or such intermediate tools
> > >>
> > >> Thanking you in advance
> > >>
> > >>
> >
> >
>
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by sunil chaudhari <su...@gmail.com>.
You can try this, if you know what prometheus and how its installed
configured.
https://www.confluent.io/blog/monitor-kafka-clusters-with-prometheus-grafana-and-confluent/
On Wed, 17 Aug 2022 at 2:25 AM, Peter Bukowinski <pm...@gmail.com> wrote:
> Richard recently answered your query. A kafka cluster does not keep track
> of lag on behalf of external consumers and it therefore is not available in
> JMX. This is why tools like Burrow were written. The java kafka consumer
> published consumer lag metrics, and perhaps some other third-party clients
> do, as well.
>
> > On Aug 16, 2022, at 12:05 PM, Kafka Life <li...@gmail.com> wrote:
> >
> > Hello Experts, Any info or pointers on my query please.
> >
> >
> >
> > On Mon, Aug 15, 2022 at 11:36 PM Kafka Life <li...@gmail.com>
> wrote:
> >
> >> Dear Kafka Experts
> >> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> >> versions of kafka in Grafana.
> >>
> >> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> >> kafka cluster.
> >>
> >> 2/ I had thought it is FetcherLag but it looks like it is not as per
> the
> >> link below.
> >>
> >>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> >> .
> >>
> >> Could one of you experts please guide on which JMX i should use for
> >> consumer lag apart from kafka burrow or such intermediate tools
> >>
> >> Thanking you in advance
> >>
> >>
>
>
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by Peter Bukowinski <pm...@gmail.com>.
Richard recently answered your query. A kafka cluster does not keep track of lag on behalf of external consumers and it therefore is not available in JMX. This is why tools like Burrow were written. The java kafka consumer published consumer lag metrics, and perhaps some other third-party clients do, as well.
> On Aug 16, 2022, at 12:05 PM, Kafka Life <li...@gmail.com> wrote:
>
> Hello Experts, Any info or pointers on my query please.
>
>
>
> On Mon, Aug 15, 2022 at 11:36 PM Kafka Life <li...@gmail.com> wrote:
>
>> Dear Kafka Experts
>> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
>> versions of kafka in Grafana.
>>
>> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
>> kafka cluster.
>>
>> 2/ I had thought it is FetcherLag but it looks like it is not as per the
>> link below.
>>
>> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
>> .
>>
>> Could one of you experts please guide on which JMX i should use for
>> consumer lag apart from kafka burrow or such intermediate tools
>>
>> Thanking you in advance
>>
>>
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by Kafka Life <li...@gmail.com>.
Hello Experts, Any info or pointers on my query please.
On Mon, Aug 15, 2022 at 11:36 PM Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag but it looks like it is not as per the
> link below.
>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
>
Re: Consumer Lag-Apache_kafka_JMX metrics
Posted by Kafka Life <li...@gmail.com>.
Hello Experts, Any info or pointers on my query please.
On Mon, Aug 15, 2022 at 11:36 PM Kafka Life <li...@gmail.com> wrote:
> Dear Kafka Experts
> we need to monitor the consumer lag in kafka clusters 2.5.1 and 2.8.0
> versions of kafka in Grafana.
>
> 1/ What is the correct path for JMX metrics to evaluate Consumer Lag in
> kafka cluster.
>
> 2/ I had thought it is FetcherLag but it looks like it is not as per the
> link below.
>
> https://www.instaclustr.com/support/documentation/kafka/monitoring-information/fetcher-lag-metrics/#:~:text=Aggregated%20Fetcher%20Consumer%20Lag%20This%20metric%20aggregates%20lag,in%20sync%20with%20partitions%20that%20it%20is%20replicating
> .
>
> Could one of you experts please guide on which JMX i should use for
> consumer lag apart from kafka burrow or such intermediate tools
>
> Thanking you in advance
>
>