You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by allen chan <al...@gmail.com> on 2015/11/14 01:06:03 UTC
consumer offset tool and JMX metrics do not match
Hi All,
I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
(kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
and they do not match.
ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0 for
all partitions.
I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
Has anyone seen this issue before? I am trying to use JMX to pull stats
into monitoring system
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
That thread is describing my exact experience.
The last reply is in June. Has anything changed? Are we supposed to stop
trusting JMX and try to use borrow or integrate with Consumer Offset
Checker? Shouldn't the documentation reflect this matter as I am sure
everyone will try to use JMX first since the docs say that is how to
monitor.
On Fri, Nov 20, 2015 at 2:32 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi Allen,
>
> I was referring to one of the issues here:
> http://search-hadoop.com/m/uyzND1XVyK12UNtd32/kafka+orphaned/v=threaded
> This linked thread discusses one of such issues where consumer lag was not
> reported correctly.
>
> Regards,
> Prabhjot
>
> On Sun, Nov 15, 2015 at 7:04 AM, allen chan <al...@gmail.com>
> wrote:
>
> > I believe producers / brokers / and consumers has been restarted at
> > different times.
> > What do you think the issue is?
> >
> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Did any of your brokers/consumers undergo a restart between the time
> > after
> > > you had started the consumption and until you see this issue ?
> > >
> > > Thanks,
> > > Prabhjot
> > >
> > >
> > >
> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan <
> > allen.michael.chan@gmail.com>
> > > wrote:
> > >
> > > > I also looked at this metric in JMX and it is also 0
> > > >
> > >
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
> > > >
> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > > allen.michael.chan@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs
> > JMX
> > > > >
> > > >
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > > > > and they do not match.
> > > > >
> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> shows
> > 0
> > > > for
> > > > > all partitions.
> > > > >
> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
> JMX.
> > > > >
> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
> > stats
> > > > > into monitoring system
> > > > >
> > > > >
> > > > > --
> > > > > Allen Michael Chan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > ---------------------------------------------------------
> > > "There are only 10 types of people in the world: Those who understand
> > > binary, and those who don't"
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi Allen,
I was referring to one of the issues here:
http://search-hadoop.com/m/uyzND1XVyK12UNtd32/kafka+orphaned/v=threaded
This linked thread discusses one of such issues where consumer lag was not
reported correctly.
Regards,
Prabhjot
On Sun, Nov 15, 2015 at 7:04 AM, allen chan <al...@gmail.com>
wrote:
> I believe producers / brokers / and consumers has been restarted at
> different times.
> What do you think the issue is?
>
> On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Did any of your brokers/consumers undergo a restart between the time
> after
> > you had started the consumption and until you see this issue ?
> >
> > Thanks,
> > Prabhjot
> >
> >
> >
> > On Sat, Nov 14, 2015 at 5:53 AM, allen chan <
> allen.michael.chan@gmail.com>
> > wrote:
> >
> > > I also looked at this metric in JMX and it is also 0
> > >
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
> > >
> > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > allen.michael.chan@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs
> JMX
> > > >
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > > > and they do not match.
> > > >
> > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows
> 0
> > > for
> > > > all partitions.
> > > >
> > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > > >
> > > > Has anyone seen this issue before? I am trying to use JMX to pull
> stats
> > > > into monitoring system
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > ---------------------------------------------------------
> > "There are only 10 types of people in the world: Those who understand
> > binary, and those who don't"
> >
>
>
>
> --
> Allen Michael Chan
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
Anyone can help me understand this?
On Mon, Nov 16, 2015 at 11:21 PM, allen chan <al...@gmail.com>
wrote:
> According to documentation, offsets by default are committed every 10
> secs. Shouldnt that be frequent enough that JMX would be accurate?
>
> autocommit.interval.ms10000is the frequency that the consumed offsets are
> committed to zookeeper.
>
> On Mon, Nov 16, 2015 at 3:31 PM, allen chan <al...@gmail.com>
> wrote:
>
>> So to make JMX accurate, we need to tweak the frequency of commits? What
>> setting would that be?
>>
>> On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sr...@twilio.com>
>> wrote:
>>
>>> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
>>> Osama.Abu-Obeid@morganstanley.com> wrote:
>>>
>>> > I can observe the same thing:
>>> >
>>> > - Lag values read through the Kafka consumer JMX is 0
>>> >
>>> This metric includes *uncommitted* offsets
>>>
>>> - Lag values read through kafka-run-class.sh
>>> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
>>> >
>>> * This is just the *committed* offsets
>>>
>>>
>>> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
>>> > ConsumerOffsetChecker shows a matching number.
>>> >
>>> > I am running kafka_2.10-0.8.2.1
>>> >
>>> > Osama
>>> >
>>> > -----Original Message-----
>>> > From: allen chan [mailto:allen.michael.chan@gmail.com]
>>> > Sent: Saturday, November 14, 2015 8:34 PM
>>> > To: users@kafka.apache.org
>>> > Subject: Re: consumer offset tool and JMX metrics do not match
>>> >
>>> > I believe producers / brokers / and consumers has been restarted at
>>> > different times.
>>> > What do you think the issue is?
>>> >
>>> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <
>>> prabhbharaj@gmail.com>
>>> > wrote:
>>> >
>>> > > Hi,
>>> > >
>>> > > Did any of your brokers/consumers undergo a restart between the time
>>> > > after you had started the consumption and until you see this issue ?
>>> > >
>>> > > Thanks,
>>> > > Prabhjot
>>> > >
>>> > >
>>> > >
>>> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
>>> > > <al...@gmail.com>
>>> > > wrote:
>>> > >
>>> > > > I also looked at this metric in JMX and it is also 0
>>> > > >
>>> > >
>>> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
>>> > > ash*
>>> > > >
>>> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
>>> > > allen.michael.chan@gmail.com>
>>> > > > wrote:
>>> > > >
>>> > > > > Hi All,
>>> > > > >
>>> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
>>> > > > > vs JMX
>>> > > > >
>>> > > >
>>> > >
>>> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
>>> > > h,topic=logstash_fdm,partition=*)
>>> > > > > and they do not match.
>>> > > > >
>>> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
>>> > > > > shows 0
>>> > > > for
>>> > > > > all partitions.
>>> > > > >
>>> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
>>> JMX.
>>> > > > >
>>> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
>>> > > > > stats into monitoring system
>>> > > > >
>>> > > > >
>>> > > > > --
>>> > > > > Allen Michael Chan
>>> > > > >
>>> > > >
>>> > > >
>>> > > >
>>> > > > --
>>> > > > Allen Michael Chan
>>> > > >
>>> > >
>>> > >
>>> > >
>>> > > --
>>> > > ---------------------------------------------------------
>>> > > "There are only 10 types of people in the world: Those who understand
>>> > > binary, and those who don't"
>>> > >
>>> >
>>> >
>>> >
>>> > --
>>> > Allen Michael Chan
>>> >
>>> >
>>> >
>>> >
>>> --------------------------------------------------------------------------------
>>> >
>>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>>> > opinions or views contained herein are not intended to be, and do not
>>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>>> Wall
>>> > Street Reform and Consumer Protection Act. If you have received this
>>> > communication in error, please destroy all electronic and paper
>>> copies; do
>>> > not disclose, use or act upon the information; and notify the sender
>>> > immediately. Mistransmission is not intended to waive confidentiality
>>> or
>>> > privilege. Morgan Stanley reserves the right, to the extent permitted
>>> under
>>> > applicable law, to monitor electronic communications. This message is
>>> > subject to terms available at the following link:
>>> > http://www.morganstanley.com/disclaimers. If you cannot access these
>>> > links, please notify us by reply message and we will send the contents
>>> to
>>> > you. By messaging with Morgan Stanley you consent to the foregoing.
>>>
>>
>>
>>
>> --
>> Allen Michael Chan
>>
>
>
>
> --
> Allen Michael Chan
>
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
According to documentation, offsets by default are committed every 10 secs.
Shouldnt that be frequent enough that JMX would be accurate?
autocommit.interval.ms10000is the frequency that the consumed offsets are
committed to zookeeper.
On Mon, Nov 16, 2015 at 3:31 PM, allen chan <al...@gmail.com>
wrote:
> So to make JMX accurate, we need to tweak the frequency of commits? What
> setting would that be?
>
> On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sr...@twilio.com>
> wrote:
>
>> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
>> Osama.Abu-Obeid@morganstanley.com> wrote:
>>
>> > I can observe the same thing:
>> >
>> > - Lag values read through the Kafka consumer JMX is 0
>> >
>> This metric includes *uncommitted* offsets
>>
>> - Lag values read through kafka-run-class.sh
>> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
>> >
>> * This is just the *committed* offsets
>>
>>
>> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
>> > ConsumerOffsetChecker shows a matching number.
>> >
>> > I am running kafka_2.10-0.8.2.1
>> >
>> > Osama
>> >
>> > -----Original Message-----
>> > From: allen chan [mailto:allen.michael.chan@gmail.com]
>> > Sent: Saturday, November 14, 2015 8:34 PM
>> > To: users@kafka.apache.org
>> > Subject: Re: consumer offset tool and JMX metrics do not match
>> >
>> > I believe producers / brokers / and consumers has been restarted at
>> > different times.
>> > What do you think the issue is?
>> >
>> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <prabhbharaj@gmail.com
>> >
>> > wrote:
>> >
>> > > Hi,
>> > >
>> > > Did any of your brokers/consumers undergo a restart between the time
>> > > after you had started the consumption and until you see this issue ?
>> > >
>> > > Thanks,
>> > > Prabhjot
>> > >
>> > >
>> > >
>> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
>> > > <al...@gmail.com>
>> > > wrote:
>> > >
>> > > > I also looked at this metric in JMX and it is also 0
>> > > >
>> > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
>> > > ash*
>> > > >
>> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
>> > > allen.michael.chan@gmail.com>
>> > > > wrote:
>> > > >
>> > > > > Hi All,
>> > > > >
>> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
>> > > > > vs JMX
>> > > > >
>> > > >
>> > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
>> > > h,topic=logstash_fdm,partition=*)
>> > > > > and they do not match.
>> > > > >
>> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
>> > > > > shows 0
>> > > > for
>> > > > > all partitions.
>> > > > >
>> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
>> JMX.
>> > > > >
>> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
>> > > > > stats into monitoring system
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Allen Michael Chan
>> > > > >
>> > > >
>> > > >
>> > > >
>> > > > --
>> > > > Allen Michael Chan
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > ---------------------------------------------------------
>> > > "There are only 10 types of people in the world: Those who understand
>> > > binary, and those who don't"
>> > >
>> >
>> >
>> >
>> > --
>> > Allen Michael Chan
>> >
>> >
>> >
>> >
>> --------------------------------------------------------------------------------
>> >
>> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
>> > opinions or views contained herein are not intended to be, and do not
>> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
>> Wall
>> > Street Reform and Consumer Protection Act. If you have received this
>> > communication in error, please destroy all electronic and paper copies;
>> do
>> > not disclose, use or act upon the information; and notify the sender
>> > immediately. Mistransmission is not intended to waive confidentiality or
>> > privilege. Morgan Stanley reserves the right, to the extent permitted
>> under
>> > applicable law, to monitor electronic communications. This message is
>> > subject to terms available at the following link:
>> > http://www.morganstanley.com/disclaimers. If you cannot access these
>> > links, please notify us by reply message and we will send the contents
>> to
>> > you. By messaging with Morgan Stanley you consent to the foregoing.
>>
>
>
>
> --
> Allen Michael Chan
>
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
So to make JMX accurate, we need to tweak the frequency of commits? What
setting would that be?
On Mon, Nov 16, 2015 at 8:40 AM, Scott Reynolds <sr...@twilio.com>
wrote:
> On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
> Osama.Abu-Obeid@morganstanley.com> wrote:
>
> > I can observe the same thing:
> >
> > - Lag values read through the Kafka consumer JMX is 0
> >
> This metric includes *uncommitted* offsets
>
> - Lag values read through kafka-run-class.sh
> > kafka.tools.ConsumerOffsetChecker is on average 200K-400K
> >
> * This is just the *committed* offsets
>
>
> > When the Lag value in the Kafka consumer JMX is high (for example 5M),
> > ConsumerOffsetChecker shows a matching number.
> >
> > I am running kafka_2.10-0.8.2.1
> >
> > Osama
> >
> > -----Original Message-----
> > From: allen chan [mailto:allen.michael.chan@gmail.com]
> > Sent: Saturday, November 14, 2015 8:34 PM
> > To: users@kafka.apache.org
> > Subject: Re: consumer offset tool and JMX metrics do not match
> >
> > I believe producers / brokers / and consumers has been restarted at
> > different times.
> > What do you think the issue is?
> >
> > On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
> > wrote:
> >
> > > Hi,
> > >
> > > Did any of your brokers/consumers undergo a restart between the time
> > > after you had started the consumption and until you see this issue ?
> > >
> > > Thanks,
> > > Prabhjot
> > >
> > >
> > >
> > > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
> > > <al...@gmail.com>
> > > wrote:
> > >
> > > > I also looked at this metric in JMX and it is also 0
> > > >
> > > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
> > > ash*
> > > >
> > > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > > allen.michael.chan@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi All,
> > > > >
> > > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
> > > > > vs JMX
> > > > >
> > > >
> > > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
> > > h,topic=logstash_fdm,partition=*)
> > > > > and they do not match.
> > > > >
> > > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> > > > > shows 0
> > > > for
> > > > > all partitions.
> > > > >
> > > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to
> JMX.
> > > > >
> > > > > Has anyone seen this issue before? I am trying to use JMX to pull
> > > > > stats into monitoring system
> > > > >
> > > > >
> > > > > --
> > > > > Allen Michael Chan
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > ---------------------------------------------------------
> > > "There are only 10 types of people in the world: Those who understand
> > > binary, and those who don't"
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
> >
> >
> >
> --------------------------------------------------------------------------------
> >
> > NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> > opinions or views contained herein are not intended to be, and do not
> > constitute, advice within the meaning of Section 975 of the Dodd-Frank
> Wall
> > Street Reform and Consumer Protection Act. If you have received this
> > communication in error, please destroy all electronic and paper copies;
> do
> > not disclose, use or act upon the information; and notify the sender
> > immediately. Mistransmission is not intended to waive confidentiality or
> > privilege. Morgan Stanley reserves the right, to the extent permitted
> under
> > applicable law, to monitor electronic communications. This message is
> > subject to terms available at the following link:
> > http://www.morganstanley.com/disclaimers. If you cannot access these
> > links, please notify us by reply message and we will send the contents to
> > you. By messaging with Morgan Stanley you consent to the foregoing.
>
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by Scott Reynolds <sr...@twilio.com>.
On Mon, Nov 16, 2015 at 8:27 AM, Abu-Obeid, Osama <
Osama.Abu-Obeid@morganstanley.com> wrote:
> I can observe the same thing:
>
> - Lag values read through the Kafka consumer JMX is 0
>
This metric includes *uncommitted* offsets
- Lag values read through kafka-run-class.sh
> kafka.tools.ConsumerOffsetChecker is on average 200K-400K
>
* This is just the *committed* offsets
> When the Lag value in the Kafka consumer JMX is high (for example 5M),
> ConsumerOffsetChecker shows a matching number.
>
> I am running kafka_2.10-0.8.2.1
>
> Osama
>
> -----Original Message-----
> From: allen chan [mailto:allen.michael.chan@gmail.com]
> Sent: Saturday, November 14, 2015 8:34 PM
> To: users@kafka.apache.org
> Subject: Re: consumer offset tool and JMX metrics do not match
>
> I believe producers / brokers / and consumers has been restarted at
> different times.
> What do you think the issue is?
>
> On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
> wrote:
>
> > Hi,
> >
> > Did any of your brokers/consumers undergo a restart between the time
> > after you had started the consumption and until you see this issue ?
> >
> > Thanks,
> > Prabhjot
> >
> >
> >
> > On Sat, Nov 14, 2015 at 5:53 AM, allen chan
> > <al...@gmail.com>
> > wrote:
> >
> > > I also looked at this metric in JMX and it is also 0
> > >
> > *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
> > ash*
> > >
> > > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> > allen.michael.chan@gmail.com>
> > > wrote:
> > >
> > > > Hi All,
> > > >
> > > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
> > > > vs JMX
> > > >
> > >
> > (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
> > h,topic=logstash_fdm,partition=*)
> > > > and they do not match.
> > > >
> > > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> > > > shows 0
> > > for
> > > > all partitions.
> > > >
> > > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > > >
> > > > Has anyone seen this issue before? I am trying to use JMX to pull
> > > > stats into monitoring system
> > > >
> > > >
> > > > --
> > > > Allen Michael Chan
> > > >
> > >
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > ---------------------------------------------------------
> > "There are only 10 types of people in the world: Those who understand
> > binary, and those who don't"
> >
>
>
>
> --
> Allen Michael Chan
>
>
>
> --------------------------------------------------------------------------------
>
> NOTICE: Morgan Stanley is not acting as a municipal advisor and the
> opinions or views contained herein are not intended to be, and do not
> constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall
> Street Reform and Consumer Protection Act. If you have received this
> communication in error, please destroy all electronic and paper copies; do
> not disclose, use or act upon the information; and notify the sender
> immediately. Mistransmission is not intended to waive confidentiality or
> privilege. Morgan Stanley reserves the right, to the extent permitted under
> applicable law, to monitor electronic communications. This message is
> subject to terms available at the following link:
> http://www.morganstanley.com/disclaimers. If you cannot access these
> links, please notify us by reply message and we will send the contents to
> you. By messaging with Morgan Stanley you consent to the foregoing.
RE: consumer offset tool and JMX metrics do not match
Posted by "Abu-Obeid, Osama" <Os...@morganstanley.com>.
I can observe the same thing:
- Lag values read through the Kafka consumer JMX is 0
- Lag values read through kafka-run-class.sh kafka.tools.ConsumerOffsetChecker is on average 200K-400K
When the Lag value in the Kafka consumer JMX is high (for example 5M), ConsumerOffsetChecker shows a matching number.
I am running kafka_2.10-0.8.2.1
Osama
-----Original Message-----
From: allen chan [mailto:allen.michael.chan@gmail.com]
Sent: Saturday, November 14, 2015 8:34 PM
To: users@kafka.apache.org
Subject: Re: consumer offset tool and JMX metrics do not match
I believe producers / brokers / and consumers has been restarted at different times.
What do you think the issue is?
On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi,
>
> Did any of your brokers/consumers undergo a restart between the time
> after you had started the consumption and until you see this issue ?
>
> Thanks,
> Prabhjot
>
>
>
> On Sat, Nov 14, 2015 at 5:53 AM, allen chan
> <al...@gmail.com>
> wrote:
>
> > I also looked at this metric in JMX and it is also 0
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logst
> ash*
> >
> > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> allen.michael.chan@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am comparing the output from kafka.tools.ConsumerOffsetChecker
> > > vs JMX
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstas
> h,topic=logstash_fdm,partition=*)
> > > and they do not match.
> > >
> > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX
> > > shows 0
> > for
> > > all partitions.
> > >
> > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > >
> > > Has anyone seen this issue before? I am trying to use JMX to pull
> > > stats into monitoring system
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
--
Allen Michael Chan
--------------------------------------------------------------------------------
NOTICE: Morgan Stanley is not acting as a municipal advisor and the opinions or views contained herein are not intended to be, and do not constitute, advice within the meaning of Section 975 of the Dodd-Frank Wall Street Reform and Consumer Protection Act. If you have received this communication in error, please destroy all electronic and paper copies; do not disclose, use or act upon the information; and notify the sender immediately. Mistransmission is not intended to waive confidentiality or privilege. Morgan Stanley reserves the right, to the extent permitted under applicable law, to monitor electronic communications. This message is subject to terms available at the following link: http://www.morganstanley.com/disclaimers. If you cannot access these links, please notify us by reply message and we will send the contents to you. By messaging with Morgan Stanley you consent to the foregoing.
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
I believe producers / brokers / and consumers has been restarted at
different times.
What do you think the issue is?
On Sat, Nov 14, 2015 at 2:52 AM, Prabhjot Bharaj <pr...@gmail.com>
wrote:
> Hi,
>
> Did any of your brokers/consumers undergo a restart between the time after
> you had started the consumption and until you see this issue ?
>
> Thanks,
> Prabhjot
>
>
>
> On Sat, Nov 14, 2015 at 5:53 AM, allen chan <al...@gmail.com>
> wrote:
>
> > I also looked at this metric in JMX and it is also 0
> >
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
> >
> > On Fri, Nov 13, 2015 at 4:06 PM, allen chan <
> allen.michael.chan@gmail.com>
> > wrote:
> >
> > > Hi All,
> > >
> > > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
> > >
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > > and they do not match.
> > >
> > > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0
> > for
> > > all partitions.
> > >
> > > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> > >
> > > Has anyone seen this issue before? I am trying to use JMX to pull stats
> > > into monitoring system
> > >
> > >
> > > --
> > > Allen Michael Chan
> > >
> >
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> ---------------------------------------------------------
> "There are only 10 types of people in the world: Those who understand
> binary, and those who don't"
>
--
Allen Michael Chan
Re: consumer offset tool and JMX metrics do not match
Posted by Prabhjot Bharaj <pr...@gmail.com>.
Hi,
Did any of your brokers/consumers undergo a restart between the time after
you had started the consumption and until you see this issue ?
Thanks,
Prabhjot
On Sat, Nov 14, 2015 at 5:53 AM, allen chan <al...@gmail.com>
wrote:
> I also looked at this metric in JMX and it is also 0
> *kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
>
> On Fri, Nov 13, 2015 at 4:06 PM, allen chan <al...@gmail.com>
> wrote:
>
> > Hi All,
> >
> > I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
> >
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> > and they do not match.
> >
> > ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0
> for
> > all partitions.
> >
> > I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
> >
> > Has anyone seen this issue before? I am trying to use JMX to pull stats
> > into monitoring system
> >
> >
> > --
> > Allen Michael Chan
> >
>
>
>
> --
> Allen Michael Chan
>
--
---------------------------------------------------------
"There are only 10 types of people in the world: Those who understand
binary, and those who don't"
Re: consumer offset tool and JMX metrics do not match
Posted by allen chan <al...@gmail.com>.
I also looked at this metric in JMX and it is also 0
*kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=logstash*
On Fri, Nov 13, 2015 at 4:06 PM, allen chan <al...@gmail.com>
wrote:
> Hi All,
>
> I am comparing the output from kafka.tools.ConsumerOffsetChecker vs JMX
> (kafka.server:type=FetcherLagMetrics,name=ConsumerLag,clientId=logstash,topic=logstash_fdm,partition=*)
> and they do not match.
>
> ConsumerOffsetChecker is showing ~60 Lag per partition and JMX shows 0 for
> all partitions.
>
> I am running kafka_2.11-0.8.2.2 and using jconsole to connect to JMX.
>
> Has anyone seen this issue before? I am trying to use JMX to pull stats
> into monitoring system
>
>
> --
> Allen Michael Chan
>
--
Allen Michael Chan