You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Xuyen On <xo...@ancestry.com> on 2014/01/24 23:41:27 UTC
Calcuate Consumer lag from JMX beans in Kafka 0.7.2
I would like to know what these attributes mean in my Kafka 0.7.2 brokers. I see the following in JConsole for my eventdata topic:
kafka.logs.eventdata-0
Attributes
Name // Name of partition
Size //Is this the current number of messages?
NumberofSegments // Don't know what this is
CurrentOffset //Is this the current offset of a consumer?
NumAppendedMessages // Don't know what this is
Can someone please tell me what these attributes mean? I am trying to find out if I can use these values to calculate the consumer lag ie: Size - CurrentOffset and report it to a chart or dashboard somewhere for real-time analysis.
Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Posted by Otis Gospodnetic <ot...@gmail.com>.
Hi Xuyen,
SPM for Kafka gets all this stuff already. I didn't look into whether/how
exactly one can spot consumer lag, but if you spot a way to do it, please
share.
There is a demo of SPM for Kafka if you go to
https://apps.sematext.com/demoso you can see all Kafka performance
graphs and see if any of the metrics
there show consumer lag directly or indirectly.
Otis
--
Performance Monitoring * Log Analytics * Search Analytics
Solr & Elasticsearch Support * http://sematext.com/
On Mon, Jan 27, 2014 at 12:49 PM, Xuyen On <xo...@ancestry.com> wrote:
> Oh, I should note that the built in consumer offsetchecker tool works for
> me but I was hoping to use something like jmxtrans so that I could easily
> export the data to ganglia, graphite or some other graphing tool. Jmxtrans
> was recommended from the Kafka wiki and there's another project called
> kafka-ganglia but it doesn't support 0.7.2 Kafka which is what we're using
> right now. Any suggestions to get real time reporting of consumer lag
> through a tool that can export to ganglia would be appreciated.
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Xuyen On [mailto:xon@ancestry.com]
> Sent: Monday, January 27, 2014 9:33 AM
> To: users@kafka.apache.org
> Subject: RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> Thanks for your response Joel.
>
> I am currently trying out JMXTrans to get the stats from MBean and I can
> read attributes fine but it doesn't support JMX Operations yet. What tool
> do you use for your reporting? Is there another tool that supports JMX
> operations so that I can use the getOffsetLag operation?
>
> Thanks,
>
> Xuyen
>
> -----Original Message-----
> From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
> Sent: Friday, January 24, 2014 5:48 PM
> To: users@kafka.apache.org
> Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
>
> > kafka.logs.eventdata-0
> > Attributes
> > Name // Name of partition
> > Size //Is this the
> current number of messages?
> Size -> in bytes
>
> > NumberofSegments // Don't know what this is
> Each partition contains multiple segment (files) on disk.
>
> > CurrentOffset //Is this the current
> offset of a consumer?
>
> No - it is the (byte) offset up to which we have flushed. It is unrelated
> to the current consumption point of consumers.
>
> > NumAppendedMessages // Don't know what this is
> Simple counter of number of messages sent so far to this partition.
>
> > Can someone please tell me what these attributes mean? I am trying to
> find out if I can use these values to calculate the consumer lag ie: Size -
> CurrentOffset and report it to a chart or dashboard somewhere for real-time
> analysis.
>
> So the above would be insufficient to measure consumer lag. You can use
> the consumer offset checker tool and extract the lag from its output. There
> is also an mbean operation on the consumers
> (getOffsetLag) that you can use on a per-partition basis.
>
> Thanks,
>
> Joel
>
>
>
RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Posted by Xuyen On <xo...@ancestry.com>.
Oh, I should note that the built in consumer offsetchecker tool works for me but I was hoping to use something like jmxtrans so that I could easily export the data to ganglia, graphite or some other graphing tool. Jmxtrans was recommended from the Kafka wiki and there's another project called kafka-ganglia but it doesn't support 0.7.2 Kafka which is what we're using right now. Any suggestions to get real time reporting of consumer lag through a tool that can export to ganglia would be appreciated.
Thanks,
Xuyen
-----Original Message-----
From: Xuyen On [mailto:xon@ancestry.com]
Sent: Monday, January 27, 2014 9:33 AM
To: users@kafka.apache.org
Subject: RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Thanks for your response Joel.
I am currently trying out JMXTrans to get the stats from MBean and I can read attributes fine but it doesn't support JMX Operations yet. What tool do you use for your reporting? Is there another tool that supports JMX operations so that I can use the getOffsetLag operation?
Thanks,
Xuyen
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Friday, January 24, 2014 5:48 PM
To: users@kafka.apache.org
Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
> kafka.logs.eventdata-0
> Attributes
> Name // Name of partition
> Size //Is this the current number of messages?
Size -> in bytes
> NumberofSegments // Don't know what this is
Each partition contains multiple segment (files) on disk.
> CurrentOffset //Is this the current offset of a consumer?
No - it is the (byte) offset up to which we have flushed. It is unrelated to the current consumption point of consumers.
> NumAppendedMessages // Don't know what this is
Simple counter of number of messages sent so far to this partition.
> Can someone please tell me what these attributes mean? I am trying to find out if I can use these values to calculate the consumer lag ie: Size - CurrentOffset and report it to a chart or dashboard somewhere for real-time analysis.
So the above would be insufficient to measure consumer lag. You can use the consumer offset checker tool and extract the lag from its output. There is also an mbean operation on the consumers
(getOffsetLag) that you can use on a per-partition basis.
Thanks,
Joel
RE: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Posted by Xuyen On <xo...@ancestry.com>.
Thanks for your response Joel.
I am currently trying out JMXTrans to get the stats from MBean and I can read attributes fine but it doesn't support JMX Operations yet. What tool do you use for your reporting? Is there another tool that supports JMX operations so that I can use the getOffsetLag operation?
Thanks,
Xuyen
-----Original Message-----
From: Joel Koshy [mailto:jjkoshy.w@gmail.com]
Sent: Friday, January 24, 2014 5:48 PM
To: users@kafka.apache.org
Subject: Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
> kafka.logs.eventdata-0
> Attributes
> Name // Name of partition
> Size //Is this the current number of messages?
Size -> in bytes
> NumberofSegments // Don't know what this is
Each partition contains multiple segment (files) on disk.
> CurrentOffset //Is this the current offset of a consumer?
No - it is the (byte) offset up to which we have flushed. It is unrelated to the current consumption point of consumers.
> NumAppendedMessages // Don't know what this is
Simple counter of number of messages sent so far to this partition.
> Can someone please tell me what these attributes mean? I am trying to find out if I can use these values to calculate the consumer lag ie: Size - CurrentOffset and report it to a chart or dashboard somewhere for real-time analysis.
So the above would be insufficient to measure consumer lag. You can use the consumer offset checker tool and extract the lag from its output. There is also an mbean operation on the consumers
(getOffsetLag) that you can use on a per-partition basis.
Thanks,
Joel
Re: Calcuate Consumer lag from JMX beans in Kafka 0.7.2
Posted by Joel Koshy <jj...@gmail.com>.
> kafka.logs.eventdata-0
> Attributes
> Name // Name of partition
> Size //Is this the current number of messages?
Size -> in bytes
> NumberofSegments // Don't know what this is
Each partition contains multiple segment (files) on disk.
> CurrentOffset //Is this the current offset of a consumer?
No - it is the (byte) offset up to which we have flushed. It is
unrelated to the current consumption point of consumers.
> NumAppendedMessages // Don't know what this is
Simple counter of number of messages sent so far to this partition.
> Can someone please tell me what these attributes mean? I am trying to find out if I can use these values to calculate the consumer lag ie: Size - CurrentOffset and report it to a chart or dashboard somewhere for real-time analysis.
So the above would be insufficient to measure consumer lag. You can
use the consumer offset checker tool and extract the lag from its
output. There is also an mbean operation on the consumers
(getOffsetLag) that you can use on a per-partition basis.
Thanks,
Joel