You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com> on 2016/11/14 22:16:19 UTC
Kafka 0.10 Monitoring tool
Hi there,
What is the best open source tool for Kafka monitoring mainly to check the offset lag. We tried the following tools:
1. Burrow
2. KafkaOffsetMonitor
3. Prometheus and Grafana
4. Kafka Manager
But nothing is working perfectly. Please help us on this.
Thanks
Achintya
RE: Kafka 0.10 Monitoring tool
Posted by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com>.
But we see the below error:
[2016-11-15 17:25:57,240] WARN Bootstrap broker host1:61616 disconnected (org.apache.kafka.clients.NetworkClient)
Error while executing consumer group command Request GROUP_COORDINATOR failed on brokers List(host1:61616 (id: -1 rack: null))
java.lang.RuntimeException: Request GROUP_COORDINATOR failed on brokers List(psaq1-wc.sys.comcast.net:61616 (id: -1 rack: null))
at kafka.admin.AdminClient.sendAnyNode(AdminClient.scala:67)
at kafka.admin.AdminClient.findCoordinator(AdminClient.scala:72)
at kafka.admin.AdminClient.describeGroup(AdminClient.scala:125)
at kafka.admin.AdminClient.describeConsumerGroup(AdminClient.scala:147)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describeGroup(ConsumerGroupCommand.scala:315)
at kafka.admin.ConsumerGroupCommand$ConsumerGroupService$class.describe(ConsumerGroupCommand.scala:86)
at kafka.admin.ConsumerGroupCommand$KafkaConsumerGroupService.describe(ConsumerGroupCommand.scala:303)
at kafka.admin.ConsumerGroupCommand$.main(ConsumerGroupCommand.scala:65)
at kafka.admin.ConsumerGroupCommand.main(ConsumerGroupCommand.scala)
-----Original Message-----
From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
Sent: Tuesday, November 15, 2016 11:26 AM
To: users@kafka.apache.org
Subject: Re: Kafka 0.10 Monitoring tool
If the consumer group is rebalancing or not consuming then it won't produce an output.
__
Sam Pegler
PRODUCTION ENGINEER
T. +44(0) 07 562 867 486
<http://www.infectiousmedia.com/>
3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com
This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer.
On 15 November 2016 at 15:30, Ghosh, Achintya (Contractor) < Achintya_Ghosh@comcast.com> wrote:
> Yes, we tried with this command but the problem is it does not return
> the output always. It's really weird. Sometimes it works and sometimes
> not. Any idea what could be the wrong?
>
> Thanks
> Achintya
>
> -----Original Message-----
> From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
> Sent: Tuesday, November 15, 2016 10:24 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka 0.10 Monitoring tool
>
> Have you thought of just wrapping the provided Java classes
> (kafka-consumer-groups.sh) with the programming language of your choice?
> For example...
>
>
> /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer
> --bootstrap-server
> eu1-kb-1:9092 --describe --group foo
> GROUP TOPIC PARTITION
> CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
> foo click 9 71971 71971
> 0 sarama_/10.240.0.46
> foo click 10 71699 71700
> 1 sarama_/10.240.0.46
> foo click 11 72052 72052
> 0 sarama_/10.240.0.46
>
> Can be quite easily parsed to provide whatever output you require.
>
>
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> <http://www.infectiousmedia.com/>
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately,
> and do not disclose the contents to another person, use it for any
> purpose, or store, or copy the information in any medium. Please also
> destroy and delete the message from your computer.
>
>
> On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Thank you Otis for your reply.
> >
> > Kafka Manger does not work during the high load, it shows the
> > timeout and Burrow and KafkaOffsetMonitor does not return the group
> > names properly even during the load.
> >
> > SPM is not an open source, so do you have anything opensource that
> > works in Kafka 0.10 version?
> >
> > Thanks
> > Achintya
> >
> > -----Original Message-----
> > From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> > Sent: Monday, November 14, 2016 9:25 PM
> > To: users@kafka.apache.org
> > Cc: dev@kafka.apache.org
> > Subject: Re: Kafka 0.10 Monitoring tool
> >
> > Hi,
> >
> > Why are these tools not working perfectly for you?
> > Does it *have to* be open-source? If not, Sematext SPM collects a
> > lot of Kafka metrics, with consumer lag being one of them --
> > https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> > offsets-monitoring/
> >
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> > Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> > Achintya_Ghosh@comcast.com> wrote:
> >
> > > Hi there,
> > > What is the best open source tool for Kafka monitoring mainly to
> > > check the offset lag. We tried the following tools:
> > >
> > >
> > > 1. Burrow
> > >
> > > 2. KafkaOffsetMonitor
> > >
> > > 3. Prometheus and Grafana
> > >
> > > 4. Kafka Manager
> > >
> > > But nothing is working perfectly. Please help us on this.
> > >
> > > Thanks
> > > Achintya
> > >
> > >
> >
>
Re: Kafka 0.10 Monitoring tool
Posted by Vincent Dautremont <vi...@olamobile.com>.
on my side I've seen this kind of error while using* ./kafka-consumer-groups.sh
--new-consumer*
from* kafka_2.11-0.10.1.0/bin *on a kafka cluster that was from a pre*
10.1.0 *version
Error while executing consumer group command Request GROUP_COORDINATOR
> failed on brokers List(kazoo002.olaasia.prv:9092 (id: -1 rack: null))
perhaps make sure you use the same tool version as thw one of the cluster
you are querying ... ?
On Wed, Nov 16, 2016 at 11:18 PM, Ghosh, Achintya (Contractor) <
Achintya_Ghosh@comcast.com> wrote:
> But most of the time I get this error:
>
> Error while executing consumer group command null
> org.apache.kafka.common.errors.DisconnectException
>
> Any help is appreciable?
>
>
> Thanks
> Achintya
>
> -----Original Message-----
> From: Vincent Dautremont [mailto:vincent.dautremont@olamobile.com]
> Sent: Wednesday, November 16, 2016 3:48 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka 0.10 Monitoring tool
>
> Just a note on that matter Sam :
> http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%
> 3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.
> gmail.com%3E
>
> On Tue, Nov 15, 2016 at 5:26 PM, Sam Pegler <sam.pegler@infectiousmedia.
> com>
> wrote:
>
> > If the consumer group is rebalancing or not consuming then it won't
> > produce an output.
> >
> > __
> >
> > Sam Pegler
> >
> > PRODUCTION ENGINEER
> >
> > T. +44(0) 07 562 867 486
> >
> > <http://www.infectiousmedia.com/>
> > 3-7 Herbal Hill / London / EC1R 5EJ
> > www.infectiousmedia.com
> >
> > This email and any attachments are confidential and may also be
> privileged.
> > If you
> > are not the intended recipient, please notify the sender immediately,
> > and do not disclose the contents to another person, use it for any
> > purpose, or store, or copy the information in any medium. Please also
> > destroy and delete the message from your computer.
> >
> >
> > On 15 November 2016 at 15:30, Ghosh, Achintya (Contractor) <
> > Achintya_Ghosh@comcast.com> wrote:
> >
> > > Yes, we tried with this command but the problem is it does not
> > > return the output always. It's really weird. Sometimes it works and
> sometimes not.
> > Any
> > > idea what could be the wrong?
> > >
> > > Thanks
> > > Achintya
> > >
> > > -----Original Message-----
> > > From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
> > > Sent: Tuesday, November 15, 2016 10:24 AM
> > > To: users@kafka.apache.org
> > > Subject: Re: Kafka 0.10 Monitoring tool
> > >
> > > Have you thought of just wrapping the provided Java classes
> > > (kafka-consumer-groups.sh) with the programming language of your
> choice?
> > > For example...
> > >
> > >
> > > /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer
> > --bootstrap-server
> > > eu1-kb-1:9092 --describe --group foo
> > > GROUP TOPIC PARTITION
> > > CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
> > > foo click 9 71971
> > 71971
> > > 0 sarama_/10.240.0.46
> > > foo click 10 71699
> > 71700
> > > 1 sarama_/10.240.0.46
> > > foo click 11 72052
> > 72052
> > > 0 sarama_/10.240.0.46
> > >
> > > Can be quite easily parsed to provide whatever output you require.
> > >
> > >
> > >
> > > __
> > >
> > > Sam Pegler
> > >
> > > PRODUCTION ENGINEER
> > >
> > > T. +44(0) 07 562 867 486
> > >
> > > <http://www.infectiousmedia.com/>
> > > 3-7 Herbal Hill / London / EC1R 5EJ
> > > www.infectiousmedia.com
> > >
> > > This email and any attachments are confidential and may also be
> > privileged.
> > > If you
> > > are not the intended recipient, please notify the sender
> > > immediately, and do not disclose the contents to another person, use
> > > it for any purpose,
> > or
> > > store, or copy the information in any medium. Please also destroy
> > > and delete the message from your computer.
> > >
> > >
> > > On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
> > > Achintya_Ghosh@comcast.com> wrote:
> > >
> > > > Thank you Otis for your reply.
> > > >
> > > > Kafka Manger does not work during the high load, it shows the
> > > > timeout and Burrow and KafkaOffsetMonitor does not return the
> > > > group names properly even during the load.
> > > >
> > > > SPM is not an open source, so do you have anything opensource that
> > > > works in Kafka 0.10 version?
> > > >
> > > > Thanks
> > > > Achintya
> > > >
> > > > -----Original Message-----
> > > > From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> > > > Sent: Monday, November 14, 2016 9:25 PM
> > > > To: users@kafka.apache.org
> > > > Cc: dev@kafka.apache.org
> > > > Subject: Re: Kafka 0.10 Monitoring tool
> > > >
> > > > Hi,
> > > >
> > > > Why are these tools not working perfectly for you?
> > > > Does it *have to* be open-source? If not, Sematext SPM collects a
> > > > lot of Kafka metrics, with consumer lag being one of them --
> > > > https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> > > > offsets-monitoring/
> > > >
> > > > Otis
> > > > --
> > > > Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> > > > Elasticsearch Consulting Support Training - http://sematext.com/
> > > >
> > > >
> > > > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> > > > Achintya_Ghosh@comcast.com> wrote:
> > > >
> > > > > Hi there,
> > > > > What is the best open source tool for Kafka monitoring mainly to
> > > > > check the offset lag. We tried the following tools:
> > > > >
> > > > >
> > > > > 1. Burrow
> > > > >
> > > > > 2. KafkaOffsetMonitor
> > > > >
> > > > > 3. Prometheus and Grafana
> > > > >
> > > > > 4. Kafka Manager
> > > > >
> > > > > But nothing is working perfectly. Please help us on this.
> > > > >
> > > > > Thanks
> > > > > Achintya
> > > > >
> > > > >
> > > >
> > >
> >
>
> --
> The information transmitted is intended only for the person or entity to
> which it is addressed and may contain confidential and/or privileged
> material. Any review, retransmission, dissemination or other use of, or
> taking of any action in reliance upon, this information by persons or
> entities other than the intended recipient is prohibited. If you received
> this in error, please contact the sender and delete the material from any
> computer.
>
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
RE: Kafka 0.10 Monitoring tool
Posted by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com>.
But most of the time I get this error:
Error while executing consumer group command null
org.apache.kafka.common.errors.DisconnectException
Any help is appreciable?
Thanks
Achintya
-----Original Message-----
From: Vincent Dautremont [mailto:vincent.dautremont@olamobile.com]
Sent: Wednesday, November 16, 2016 3:48 AM
To: users@kafka.apache.org
Subject: Re: Kafka 0.10 Monitoring tool
Just a note on that matter Sam :
http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.gmail.com%3E
On Tue, Nov 15, 2016 at 5:26 PM, Sam Pegler <sa...@infectiousmedia.com>
wrote:
> If the consumer group is rebalancing or not consuming then it won't
> produce an output.
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> <http://www.infectiousmedia.com/>
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately,
> and do not disclose the contents to another person, use it for any
> purpose, or store, or copy the information in any medium. Please also
> destroy and delete the message from your computer.
>
>
> On 15 November 2016 at 15:30, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Yes, we tried with this command but the problem is it does not
> > return the output always. It's really weird. Sometimes it works and sometimes not.
> Any
> > idea what could be the wrong?
> >
> > Thanks
> > Achintya
> >
> > -----Original Message-----
> > From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
> > Sent: Tuesday, November 15, 2016 10:24 AM
> > To: users@kafka.apache.org
> > Subject: Re: Kafka 0.10 Monitoring tool
> >
> > Have you thought of just wrapping the provided Java classes
> > (kafka-consumer-groups.sh) with the programming language of your choice?
> > For example...
> >
> >
> > /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer
> --bootstrap-server
> > eu1-kb-1:9092 --describe --group foo
> > GROUP TOPIC PARTITION
> > CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
> > foo click 9 71971
> 71971
> > 0 sarama_/10.240.0.46
> > foo click 10 71699
> 71700
> > 1 sarama_/10.240.0.46
> > foo click 11 72052
> 72052
> > 0 sarama_/10.240.0.46
> >
> > Can be quite easily parsed to provide whatever output you require.
> >
> >
> >
> > __
> >
> > Sam Pegler
> >
> > PRODUCTION ENGINEER
> >
> > T. +44(0) 07 562 867 486
> >
> > <http://www.infectiousmedia.com/>
> > 3-7 Herbal Hill / London / EC1R 5EJ
> > www.infectiousmedia.com
> >
> > This email and any attachments are confidential and may also be
> privileged.
> > If you
> > are not the intended recipient, please notify the sender
> > immediately, and do not disclose the contents to another person, use
> > it for any purpose,
> or
> > store, or copy the information in any medium. Please also destroy
> > and delete the message from your computer.
> >
> >
> > On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
> > Achintya_Ghosh@comcast.com> wrote:
> >
> > > Thank you Otis for your reply.
> > >
> > > Kafka Manger does not work during the high load, it shows the
> > > timeout and Burrow and KafkaOffsetMonitor does not return the
> > > group names properly even during the load.
> > >
> > > SPM is not an open source, so do you have anything opensource that
> > > works in Kafka 0.10 version?
> > >
> > > Thanks
> > > Achintya
> > >
> > > -----Original Message-----
> > > From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> > > Sent: Monday, November 14, 2016 9:25 PM
> > > To: users@kafka.apache.org
> > > Cc: dev@kafka.apache.org
> > > Subject: Re: Kafka 0.10 Monitoring tool
> > >
> > > Hi,
> > >
> > > Why are these tools not working perfectly for you?
> > > Does it *have to* be open-source? If not, Sematext SPM collects a
> > > lot of Kafka metrics, with consumer lag being one of them --
> > > https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> > > offsets-monitoring/
> > >
> > > Otis
> > > --
> > > Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> > > Elasticsearch Consulting Support Training - http://sematext.com/
> > >
> > >
> > > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> > > Achintya_Ghosh@comcast.com> wrote:
> > >
> > > > Hi there,
> > > > What is the best open source tool for Kafka monitoring mainly to
> > > > check the offset lag. We tried the following tools:
> > > >
> > > >
> > > > 1. Burrow
> > > >
> > > > 2. KafkaOffsetMonitor
> > > >
> > > > 3. Prometheus and Grafana
> > > >
> > > > 4. Kafka Manager
> > > >
> > > > But nothing is working perfectly. Please help us on this.
> > > >
> > > > Thanks
> > > > Achintya
> > > >
> > > >
> > >
> >
>
--
The information transmitted is intended only for the person or entity to which it is addressed and may contain confidential and/or privileged material. Any review, retransmission, dissemination or other use of, or taking of any action in reliance upon, this information by persons or entities other than the intended recipient is prohibited. If you received this in error, please contact the sender and delete the material from any computer.
Re: Kafka 0.10 Monitoring tool
Posted by Vincent Dautremont <vi...@olamobile.com>.
Just a note on that matter Sam :
http://mail-archives.apache.org/mod_mbox/kafka-users/201611.mbox/%3CCAD2WViSAgwc9i4-9xEw1oz1xzpsbveFt1%3DSZ0qkHRiFEc3fXbw%40mail.gmail.com%3E
On Tue, Nov 15, 2016 at 5:26 PM, Sam Pegler <sa...@infectiousmedia.com>
wrote:
> If the consumer group is rebalancing or not consuming then it won't produce
> an output.
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> <http://www.infectiousmedia.com/>
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately, and
> do not
> disclose the contents to another person, use it for any purpose, or store,
> or copy
> the information in any medium. Please also destroy and delete the message
> from
> your computer.
>
>
> On 15 November 2016 at 15:30, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Yes, we tried with this command but the problem is it does not return the
> > output always. It's really weird. Sometimes it works and sometimes not.
> Any
> > idea what could be the wrong?
> >
> > Thanks
> > Achintya
> >
> > -----Original Message-----
> > From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
> > Sent: Tuesday, November 15, 2016 10:24 AM
> > To: users@kafka.apache.org
> > Subject: Re: Kafka 0.10 Monitoring tool
> >
> > Have you thought of just wrapping the provided Java classes
> > (kafka-consumer-groups.sh) with the programming language of your choice?
> > For example...
> >
> >
> > /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer
> --bootstrap-server
> > eu1-kb-1:9092 --describe --group foo
> > GROUP TOPIC PARTITION
> > CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
> > foo click 9 71971
> 71971
> > 0 sarama_/10.240.0.46
> > foo click 10 71699
> 71700
> > 1 sarama_/10.240.0.46
> > foo click 11 72052
> 72052
> > 0 sarama_/10.240.0.46
> >
> > Can be quite easily parsed to provide whatever output you require.
> >
> >
> >
> > __
> >
> > Sam Pegler
> >
> > PRODUCTION ENGINEER
> >
> > T. +44(0) 07 562 867 486
> >
> > <http://www.infectiousmedia.com/>
> > 3-7 Herbal Hill / London / EC1R 5EJ
> > www.infectiousmedia.com
> >
> > This email and any attachments are confidential and may also be
> privileged.
> > If you
> > are not the intended recipient, please notify the sender immediately, and
> > do not disclose the contents to another person, use it for any purpose,
> or
> > store, or copy the information in any medium. Please also destroy and
> > delete the message from your computer.
> >
> >
> > On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
> > Achintya_Ghosh@comcast.com> wrote:
> >
> > > Thank you Otis for your reply.
> > >
> > > Kafka Manger does not work during the high load, it shows the timeout
> > > and Burrow and KafkaOffsetMonitor does not return the group names
> > > properly even during the load.
> > >
> > > SPM is not an open source, so do you have anything opensource that
> > > works in Kafka 0.10 version?
> > >
> > > Thanks
> > > Achintya
> > >
> > > -----Original Message-----
> > > From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> > > Sent: Monday, November 14, 2016 9:25 PM
> > > To: users@kafka.apache.org
> > > Cc: dev@kafka.apache.org
> > > Subject: Re: Kafka 0.10 Monitoring tool
> > >
> > > Hi,
> > >
> > > Why are these tools not working perfectly for you?
> > > Does it *have to* be open-source? If not, Sematext SPM collects a lot
> > > of Kafka metrics, with consumer lag being one of them --
> > > https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> > > offsets-monitoring/
> > >
> > > Otis
> > > --
> > > Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> > > Elasticsearch Consulting Support Training - http://sematext.com/
> > >
> > >
> > > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> > > Achintya_Ghosh@comcast.com> wrote:
> > >
> > > > Hi there,
> > > > What is the best open source tool for Kafka monitoring mainly to
> > > > check the offset lag. We tried the following tools:
> > > >
> > > >
> > > > 1. Burrow
> > > >
> > > > 2. KafkaOffsetMonitor
> > > >
> > > > 3. Prometheus and Grafana
> > > >
> > > > 4. Kafka Manager
> > > >
> > > > But nothing is working perfectly. Please help us on this.
> > > >
> > > > Thanks
> > > > Achintya
> > > >
> > > >
> > >
> >
>
--
The information transmitted is intended only for the person or entity to
which it is addressed and may contain confidential and/or privileged
material. Any review, retransmission, dissemination or other use of, or
taking of any action in reliance upon, this information by persons or
entities other than the intended recipient is prohibited. If you received
this in error, please contact the sender and delete the material from any
computer.
Re: Kafka 0.10 Monitoring tool
Posted by Sam Pegler <sa...@infectiousmedia.com>.
If the consumer group is rebalancing or not consuming then it won't produce
an output.
__
Sam Pegler
PRODUCTION ENGINEER
T. +44(0) 07 562 867 486
<http://www.infectiousmedia.com/>
3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com
This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.
On 15 November 2016 at 15:30, Ghosh, Achintya (Contractor) <
Achintya_Ghosh@comcast.com> wrote:
> Yes, we tried with this command but the problem is it does not return the
> output always. It's really weird. Sometimes it works and sometimes not. Any
> idea what could be the wrong?
>
> Thanks
> Achintya
>
> -----Original Message-----
> From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
> Sent: Tuesday, November 15, 2016 10:24 AM
> To: users@kafka.apache.org
> Subject: Re: Kafka 0.10 Monitoring tool
>
> Have you thought of just wrapping the provided Java classes
> (kafka-consumer-groups.sh) with the programming language of your choice?
> For example...
>
>
> /opt/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
> eu1-kb-1:9092 --describe --group foo
> GROUP TOPIC PARTITION
> CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
> foo click 9 71971 71971
> 0 sarama_/10.240.0.46
> foo click 10 71699 71700
> 1 sarama_/10.240.0.46
> foo click 11 72052 72052
> 0 sarama_/10.240.0.46
>
> Can be quite easily parsed to provide whatever output you require.
>
>
>
> __
>
> Sam Pegler
>
> PRODUCTION ENGINEER
>
> T. +44(0) 07 562 867 486
>
> <http://www.infectiousmedia.com/>
> 3-7 Herbal Hill / London / EC1R 5EJ
> www.infectiousmedia.com
>
> This email and any attachments are confidential and may also be privileged.
> If you
> are not the intended recipient, please notify the sender immediately, and
> do not disclose the contents to another person, use it for any purpose, or
> store, or copy the information in any medium. Please also destroy and
> delete the message from your computer.
>
>
> On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Thank you Otis for your reply.
> >
> > Kafka Manger does not work during the high load, it shows the timeout
> > and Burrow and KafkaOffsetMonitor does not return the group names
> > properly even during the load.
> >
> > SPM is not an open source, so do you have anything opensource that
> > works in Kafka 0.10 version?
> >
> > Thanks
> > Achintya
> >
> > -----Original Message-----
> > From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> > Sent: Monday, November 14, 2016 9:25 PM
> > To: users@kafka.apache.org
> > Cc: dev@kafka.apache.org
> > Subject: Re: Kafka 0.10 Monitoring tool
> >
> > Hi,
> >
> > Why are these tools not working perfectly for you?
> > Does it *have to* be open-source? If not, Sematext SPM collects a lot
> > of Kafka metrics, with consumer lag being one of them --
> > https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> > offsets-monitoring/
> >
> > Otis
> > --
> > Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> > Elasticsearch Consulting Support Training - http://sematext.com/
> >
> >
> > On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> > Achintya_Ghosh@comcast.com> wrote:
> >
> > > Hi there,
> > > What is the best open source tool for Kafka monitoring mainly to
> > > check the offset lag. We tried the following tools:
> > >
> > >
> > > 1. Burrow
> > >
> > > 2. KafkaOffsetMonitor
> > >
> > > 3. Prometheus and Grafana
> > >
> > > 4. Kafka Manager
> > >
> > > But nothing is working perfectly. Please help us on this.
> > >
> > > Thanks
> > > Achintya
> > >
> > >
> >
>
RE: Kafka 0.10 Monitoring tool
Posted by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com>.
Yes, we tried with this command but the problem is it does not return the output always. It's really weird. Sometimes it works and sometimes not. Any idea what could be the wrong?
Thanks
Achintya
-----Original Message-----
From: Sam Pegler [mailto:sam.pegler@infectiousmedia.com]
Sent: Tuesday, November 15, 2016 10:24 AM
To: users@kafka.apache.org
Subject: Re: Kafka 0.10 Monitoring tool
Have you thought of just wrapping the provided Java classes
(kafka-consumer-groups.sh) with the programming language of your choice?
For example...
/opt/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
eu1-kb-1:9092 --describe --group foo
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
foo click 9 71971 71971
0 sarama_/10.240.0.46
foo click 10 71699 71700
1 sarama_/10.240.0.46
foo click 11 72052 72052
0 sarama_/10.240.0.46
Can be quite easily parsed to provide whatever output you require.
__
Sam Pegler
PRODUCTION ENGINEER
T. +44(0) 07 562 867 486
<http://www.infectiousmedia.com/>
3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com
This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and do not disclose the contents to another person, use it for any purpose, or store, or copy the information in any medium. Please also destroy and delete the message from your computer.
On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) < Achintya_Ghosh@comcast.com> wrote:
> Thank you Otis for your reply.
>
> Kafka Manger does not work during the high load, it shows the timeout
> and Burrow and KafkaOffsetMonitor does not return the group names
> properly even during the load.
>
> SPM is not an open source, so do you have anything opensource that
> works in Kafka 0.10 version?
>
> Thanks
> Achintya
>
> -----Original Message-----
> From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> Sent: Monday, November 14, 2016 9:25 PM
> To: users@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka 0.10 Monitoring tool
>
> Hi,
>
> Why are these tools not working perfectly for you?
> Does it *have to* be open-source? If not, Sematext SPM collects a lot
> of Kafka metrics, with consumer lag being one of them --
> https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> offsets-monitoring/
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Hi there,
> > What is the best open source tool for Kafka monitoring mainly to
> > check the offset lag. We tried the following tools:
> >
> >
> > 1. Burrow
> >
> > 2. KafkaOffsetMonitor
> >
> > 3. Prometheus and Grafana
> >
> > 4. Kafka Manager
> >
> > But nothing is working perfectly. Please help us on this.
> >
> > Thanks
> > Achintya
> >
> >
>
Re: Kafka 0.10 Monitoring tool
Posted by Sam Pegler <sa...@infectiousmedia.com>.
Have you thought of just wrapping the provided Java classes
(kafka-consumer-groups.sh) with the programming language of your choice?
For example...
/opt/kafka/bin/kafka-consumer-groups.sh --new-consumer --bootstrap-server
eu1-kb-1:9092 --describe --group foo
GROUP TOPIC PARTITION
CURRENT-OFFSET LOG-END-OFFSET LAG OWNER
foo click 9 71971 71971
0 sarama_/10.240.0.46
foo click 10 71699 71700
1 sarama_/10.240.0.46
foo click 11 72052 72052
0 sarama_/10.240.0.46
Can be quite easily parsed to provide whatever output you require.
__
Sam Pegler
PRODUCTION ENGINEER
T. +44(0) 07 562 867 486
<http://www.infectiousmedia.com/>
3-7 Herbal Hill / London / EC1R 5EJ
www.infectiousmedia.com
This email and any attachments are confidential and may also be privileged.
If you
are not the intended recipient, please notify the sender immediately, and
do not
disclose the contents to another person, use it for any purpose, or store,
or copy
the information in any medium. Please also destroy and delete the message
from
your computer.
On 15 November 2016 at 14:28, Ghosh, Achintya (Contractor) <
Achintya_Ghosh@comcast.com> wrote:
> Thank you Otis for your reply.
>
> Kafka Manger does not work during the high load, it shows the timeout and
> Burrow and KafkaOffsetMonitor does not return the group names properly even
> during the load.
>
> SPM is not an open source, so do you have anything opensource that works
> in Kafka 0.10 version?
>
> Thanks
> Achintya
>
> -----Original Message-----
> From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
> Sent: Monday, November 14, 2016 9:25 PM
> To: users@kafka.apache.org
> Cc: dev@kafka.apache.org
> Subject: Re: Kafka 0.10 Monitoring tool
>
> Hi,
>
> Why are these tools not working perfectly for you?
> Does it *have to* be open-source? If not, Sematext SPM collects a lot of
> Kafka metrics, with consumer lag being one of them --
> https://sematext.com/blog/2016/06/07/kafka-consumer-lag-
> offsets-monitoring/
>
> Otis
> --
> Monitoring - Log Management - Alerting - Anomaly Detection Solr &
> Elasticsearch Consulting Support Training - http://sematext.com/
>
>
> On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
> Achintya_Ghosh@comcast.com> wrote:
>
> > Hi there,
> > What is the best open source tool for Kafka monitoring mainly to check
> > the offset lag. We tried the following tools:
> >
> >
> > 1. Burrow
> >
> > 2. KafkaOffsetMonitor
> >
> > 3. Prometheus and Grafana
> >
> > 4. Kafka Manager
> >
> > But nothing is working perfectly. Please help us on this.
> >
> > Thanks
> > Achintya
> >
> >
>
RE: Kafka 0.10 Monitoring tool
Posted by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com>.
Thank you Otis for your reply.
Kafka Manger does not work during the high load, it shows the timeout and Burrow and KafkaOffsetMonitor does not return the group names properly even during the load.
SPM is not an open source, so do you have anything opensource that works in Kafka 0.10 version?
Thanks
Achintya
-----Original Message-----
From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
Sent: Monday, November 14, 2016 9:25 PM
To: users@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka 0.10 Monitoring tool
Hi,
Why are these tools not working perfectly for you?
Does it *have to* be open-source? If not, Sematext SPM collects a lot of Kafka metrics, with consumer lag being one of them -- https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) < Achintya_Ghosh@comcast.com> wrote:
> Hi there,
> What is the best open source tool for Kafka monitoring mainly to check
> the offset lag. We tried the following tools:
>
>
> 1. Burrow
>
> 2. KafkaOffsetMonitor
>
> 3. Prometheus and Grafana
>
> 4. Kafka Manager
>
> But nothing is working perfectly. Please help us on this.
>
> Thanks
> Achintya
>
>
RE: Kafka 0.10 Monitoring tool
Posted by "Ghosh, Achintya (Contractor)" <Ac...@comcast.com>.
Thank you Otis for your reply.
Kafka Manger does not work during the high load, it shows the timeout and Burrow and KafkaOffsetMonitor does not return the group names properly even during the load.
SPM is not an open source, so do you have anything opensource that works in Kafka 0.10 version?
Thanks
Achintya
-----Original Message-----
From: Otis Gospodnetić [mailto:otis.gospodnetic@gmail.com]
Sent: Monday, November 14, 2016 9:25 PM
To: users@kafka.apache.org
Cc: dev@kafka.apache.org
Subject: Re: Kafka 0.10 Monitoring tool
Hi,
Why are these tools not working perfectly for you?
Does it *have to* be open-source? If not, Sematext SPM collects a lot of Kafka metrics, with consumer lag being one of them -- https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) < Achintya_Ghosh@comcast.com> wrote:
> Hi there,
> What is the best open source tool for Kafka monitoring mainly to check
> the offset lag. We tried the following tools:
>
>
> 1. Burrow
>
> 2. KafkaOffsetMonitor
>
> 3. Prometheus and Grafana
>
> 4. Kafka Manager
>
> But nothing is working perfectly. Please help us on this.
>
> Thanks
> Achintya
>
>
Re: Kafka 0.10 Monitoring tool
Posted by Otis Gospodnetić <ot...@gmail.com>.
Hi,
Why are these tools not working perfectly for you?
Does it *have to* be open-source? If not, Sematext SPM collects a lot of
Kafka metrics, with consumer lag being one of them --
https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
Achintya_Ghosh@comcast.com> wrote:
> Hi there,
> What is the best open source tool for Kafka monitoring mainly to check the
> offset lag. We tried the following tools:
>
>
> 1. Burrow
>
> 2. KafkaOffsetMonitor
>
> 3. Prometheus and Grafana
>
> 4. Kafka Manager
>
> But nothing is working perfectly. Please help us on this.
>
> Thanks
> Achintya
>
>
Re: Kafka 0.10 Monitoring tool
Posted by Otis Gospodnetić <ot...@gmail.com>.
Hi,
Why are these tools not working perfectly for you?
Does it *have to* be open-source? If not, Sematext SPM collects a lot of
Kafka metrics, with consumer lag being one of them --
https://sematext.com/blog/2016/06/07/kafka-consumer-lag-offsets-monitoring/
Otis
--
Monitoring - Log Management - Alerting - Anomaly Detection
Solr & Elasticsearch Consulting Support Training - http://sematext.com/
On Mon, Nov 14, 2016 at 5:16 PM, Ghosh, Achintya (Contractor) <
Achintya_Ghosh@comcast.com> wrote:
> Hi there,
> What is the best open source tool for Kafka monitoring mainly to check the
> offset lag. We tried the following tools:
>
>
> 1. Burrow
>
> 2. KafkaOffsetMonitor
>
> 3. Prometheus and Grafana
>
> 4. Kafka Manager
>
> But nothing is working perfectly. Please help us on this.
>
> Thanks
> Achintya
>
>