You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by nitin sharma <ku...@gmail.com> on 2014/12/09 22:26:59 UTC
How to raise a question in forum
HI,
I need some help related to Kafka Consumer Lag monitoring.. Kindly let me
know how can i post my query to the forum.
Regards,
Nitin Kumar Sharma.
Re: How to raise a question in forum
Posted by nitin sharma <ku...@gmail.com>.
Hello Otis,
actually my operation team told me that they are seeing following JMX MBean
on one of the broker.
" kafka.server* >* ReplicaFetcherThread-0-2-host_*<<KafkaHost>>*-port_
*<KafkaPort>*-*<<TopicName>>*-*<<PartitionID>>*-ConsumerLag
this is what confusing me.. and when i monitored this Mbean, i saw values
coming in negative.
Regards,
Nitin Kumar Sharma.
On Wed, Dec 10, 2014 at 4:34 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:
> Hi,
>
> On Wed, Dec 10, 2014 at 12:02 AM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> wrote:
>
> > thanks for response. But my questions are still not answered
> >
> > a. Is ConsumerLag is the metrics that is exposed on kafka broker end or
> on
> > Consumer side
> >
>
> Consumer.
>
>
> > b. Can ConsumerLag value be in negative?
> >
>
> Shouldn't be. Not sure what the meaning of that would be? You see a
> negative value somewhere?
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
>
> > On Tue, Dec 9, 2014 at 5:23 PM, Otis Gospodnetic <
> > otis.gospodnetic@gmail.com
> > > wrote:
> >
> > > Nitin,
> > >
> > > If you don't want to DIY there are already tools and solutions for
> > > monitoring Kafka, including consumer lag. Ours is mentioned here:
> > >
> > >
> >
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> > > Others have been mentioned on the list a few times before:
> > > http://search-hadoop.com/?q=monitor+lag&fc_project=Kafka
> > >
> > > Otis
> > > --
> > > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > > Solr & Elasticsearch Support * http://sematext.com/
> > >
> > >
> > > On Tue, Dec 9, 2014 at 4:59 PM, nitin sharma <
> > kumarsharma.nitin@gmail.com>
> > > wrote:
> > >
> > > > Hi Neha, Guozhang Wang
> > > >
> > > > My project requirement is to monitor Consumer Lag Mbean..
> > > >
> > > > 1) According to me this MBean is exposed only at the Consumer side
> and
> > > not
> > > > on Broker server. This is because in my initial PoC, i had used
> > > > "kafka-consumer-perf-test.sh" and MBean exposed by this consumer had
> > > > "ConsumerLag" counter.
> > > >
> > > > 2) ConsumerLag value can not be in negative. it has to be either zero
> > or
> > > > positive. is this a correct assumption? i found the following link on
> > > > internet where some one had got a negative value :
> > > >
> > > >
> > > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCA+TtWKMbJhy_q+io3_7XSQXxfc6hhF7Pzap5SFgtZd22stbjtA@mail.gmail.com%3E
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > > >
> > > > On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang <wa...@gmail.com>
> > > wrote:
> > > >
> > > > > Nitin,
> > > > >
> > > > > You have found the place :) Just send an email to users@ mailing
> > list
> > > is
> > > > > fine.
> > > > >
> > > > > Guozhang
> > > > >
> > > > > On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <
> > > > kumarsharma.nitin@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > HI,
> > > > > > I need some help related to Kafka Consumer Lag monitoring..
> Kindly
> > > let
> > > > me
> > > > > > know how can i post my query to the forum.
> > > > > >
> > > > > > Regards,
> > > > > > Nitin Kumar Sharma.
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > -- Guozhang
> > > > >
> > > >
> > >
> >
>
Re: How to raise a question in forum
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi,
On Wed, Dec 10, 2014 at 12:02 AM, nitin sharma <ku...@gmail.com>
wrote:
> thanks for response. But my questions are still not answered
>
> a. Is ConsumerLag is the metrics that is exposed on kafka broker end or on
> Consumer side
>
Consumer.
> b. Can ConsumerLag value be in negative?
>
Shouldn't be. Not sure what the meaning of that would be? You see a
negative value somewhere?
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
> On Tue, Dec 9, 2014 at 5:23 PM, Otis Gospodnetic <
> otis.gospodnetic@gmail.com
> > wrote:
>
> > Nitin,
> >
> > If you don't want to DIY there are already tools and solutions for
> > monitoring Kafka, including consumer lag. Ours is mentioned here:
> >
> >
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> > Others have been mentioned on the list a few times before:
> > http://search-hadoop.com/?q=monitor+lag&fc_project=Kafka
> >
> > Otis
> > --
> > Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> > Solr & Elasticsearch Support * http://sematext.com/
> >
> >
> > On Tue, Dec 9, 2014 at 4:59 PM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > Hi Neha, Guozhang Wang
> > >
> > > My project requirement is to monitor Consumer Lag Mbean..
> > >
> > > 1) According to me this MBean is exposed only at the Consumer side and
> > not
> > > on Broker server. This is because in my initial PoC, i had used
> > > "kafka-consumer-perf-test.sh" and MBean exposed by this consumer had
> > > "ConsumerLag" counter.
> > >
> > > 2) ConsumerLag value can not be in negative. it has to be either zero
> or
> > > positive. is this a correct assumption? i found the following link on
> > > internet where some one had got a negative value :
> > >
> > >
> > >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCA+TtWKMbJhy_q+io3_7XSQXxfc6hhF7Pzap5SFgtZd22stbjtA@mail.gmail.com%3E
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> > >
> > > On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang <wa...@gmail.com>
> > wrote:
> > >
> > > > Nitin,
> > > >
> > > > You have found the place :) Just send an email to users@ mailing
> list
> > is
> > > > fine.
> > > >
> > > > Guozhang
> > > >
> > > > On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <
> > > kumarsharma.nitin@gmail.com>
> > > > wrote:
> > > >
> > > > > HI,
> > > > > I need some help related to Kafka Consumer Lag monitoring.. Kindly
> > let
> > > me
> > > > > know how can i post my query to the forum.
> > > > >
> > > > > Regards,
> > > > > Nitin Kumar Sharma.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > -- Guozhang
> > > >
> > >
> >
>
Re: How to raise a question in forum
Posted by nitin sharma <ku...@gmail.com>.
thanks for response. But my questions are still not answered
a. Is ConsumerLag is the metrics that is exposed on kafka broker end or on
Consumer side
b. Can ConsumerLag value be in negative?
Regards,
Nitin Kumar Sharma.
On Tue, Dec 9, 2014 at 5:23 PM, Otis Gospodnetic <otis.gospodnetic@gmail.com
> wrote:
> Nitin,
>
> If you don't want to DIY there are already tools and solutions for
> monitoring Kafka, including consumer lag. Ours is mentioned here:
>
> http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
> Others have been mentioned on the list a few times before:
> http://search-hadoop.com/?q=monitor+lag&fc_project=Kafka
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Tue, Dec 9, 2014 at 4:59 PM, nitin sharma <ku...@gmail.com>
> wrote:
>
> > Hi Neha, Guozhang Wang
> >
> > My project requirement is to monitor Consumer Lag Mbean..
> >
> > 1) According to me this MBean is exposed only at the Consumer side and
> not
> > on Broker server. This is because in my initial PoC, i had used
> > "kafka-consumer-perf-test.sh" and MBean exposed by this consumer had
> > "ConsumerLag" counter.
> >
> > 2) ConsumerLag value can not be in negative. it has to be either zero or
> > positive. is this a correct assumption? i found the following link on
> > internet where some one had got a negative value :
> >
> >
> >
> http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCA+TtWKMbJhy_q+io3_7XSQXxfc6hhF7Pzap5SFgtZd22stbjtA@mail.gmail.com%3E
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
> >
> > On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang <wa...@gmail.com>
> wrote:
> >
> > > Nitin,
> > >
> > > You have found the place :) Just send an email to users@ mailing list
> is
> > > fine.
> > >
> > > Guozhang
> > >
> > > On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <
> > kumarsharma.nitin@gmail.com>
> > > wrote:
> > >
> > > > HI,
> > > > I need some help related to Kafka Consumer Lag monitoring.. Kindly
> let
> > me
> > > > know how can i post my query to the forum.
> > > >
> > > > Regards,
> > > > Nitin Kumar Sharma.
> > > >
> > >
> > >
> > >
> > > --
> > > -- Guozhang
> > >
> >
>
Re: How to raise a question in forum
Posted by Otis Gospodnetic <ot...@gmail.com>.
Nitin,
If you don't want to DIY there are already tools and solutions for
monitoring Kafka, including consumer lag. Ours is mentioned here:
http://blog.sematext.com/2013/10/16/announcement-spm-performance-monitoring-for-kafka/
Others have been mentioned on the list a few times before:
http://search-hadoop.com/?q=monitor+lag&fc_project=Kafka
Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/
On Tue, Dec 9, 2014 at 4:59 PM, nitin sharma <ku...@gmail.com>
wrote:
> Hi Neha, Guozhang Wang
>
> My project requirement is to monitor Consumer Lag Mbean..
>
> 1) According to me this MBean is exposed only at the Consumer side and not
> on Broker server. This is because in my initial PoC, i had used
> "kafka-consumer-perf-test.sh" and MBean exposed by this consumer had
> "ConsumerLag" counter.
>
> 2) ConsumerLag value can not be in negative. it has to be either zero or
> positive. is this a correct assumption? i found the following link on
> internet where some one had got a negative value :
>
>
> http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCA+TtWKMbJhy_q+io3_7XSQXxfc6hhF7Pzap5SFgtZd22stbjtA@mail.gmail.com%3E
>
> Regards,
> Nitin Kumar Sharma.
>
>
> On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang <wa...@gmail.com> wrote:
>
> > Nitin,
> >
> > You have found the place :) Just send an email to users@ mailing list is
> > fine.
> >
> > Guozhang
> >
> > On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <
> kumarsharma.nitin@gmail.com>
> > wrote:
> >
> > > HI,
> > > I need some help related to Kafka Consumer Lag monitoring.. Kindly let
> me
> > > know how can i post my query to the forum.
> > >
> > > Regards,
> > > Nitin Kumar Sharma.
> > >
> >
> >
> >
> > --
> > -- Guozhang
> >
>
Re: How to raise a question in forum
Posted by nitin sharma <ku...@gmail.com>.
Hi Neha, Guozhang Wang
My project requirement is to monitor Consumer Lag Mbean..
1) According to me this MBean is exposed only at the Consumer side and not
on Broker server. This is because in my initial PoC, i had used
"kafka-consumer-perf-test.sh" and MBean exposed by this consumer had
"ConsumerLag" counter.
2) ConsumerLag value can not be in negative. it has to be either zero or
positive. is this a correct assumption? i found the following link on
internet where some one had got a negative value :
http://mail-archives.apache.org/mod_mbox/kafka-users/201410.mbox/%3CCA+TtWKMbJhy_q+io3_7XSQXxfc6hhF7Pzap5SFgtZd22stbjtA@mail.gmail.com%3E
Regards,
Nitin Kumar Sharma.
On Tue, Dec 9, 2014 at 4:47 PM, Guozhang Wang <wa...@gmail.com> wrote:
> Nitin,
>
> You have found the place :) Just send an email to users@ mailing list is
> fine.
>
> Guozhang
>
> On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <ku...@gmail.com>
> wrote:
>
> > HI,
> > I need some help related to Kafka Consumer Lag monitoring.. Kindly let me
> > know how can i post my query to the forum.
> >
> > Regards,
> > Nitin Kumar Sharma.
> >
>
>
>
> --
> -- Guozhang
>
Re: How to raise a question in forum
Posted by Guozhang Wang <wa...@gmail.com>.
Nitin,
You have found the place :) Just send an email to users@ mailing list is
fine.
Guozhang
On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <ku...@gmail.com>
wrote:
> HI,
> I need some help related to Kafka Consumer Lag monitoring.. Kindly let me
> know how can i post my query to the forum.
>
> Regards,
> Nitin Kumar Sharma.
>
--
-- Guozhang
Re: How to raise a question in forum
Posted by Neha Narkhede <ne...@confluent.io>.
You can post it here :-)
On Tue, Dec 9, 2014 at 1:26 PM, nitin sharma <ku...@gmail.com>
wrote:
> HI,
> I need some help related to Kafka Consumer Lag monitoring.. Kindly let me
> know how can i post my query to the forum.
>
> Regards,
> Nitin Kumar Sharma.
>
--
Thanks,
Neha