You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Bhavesh Mistry <mi...@gmail.com> on 2015/02/05 20:54:00 UTC
Get Latest Offset for Specific Topic for All Partition
HI All,
I just need to get the latest offset # for topic (not for consumer group).
Which API to get this info ?
My use case is to analyze the data injection rate to each of partition is
uniform or not (close). For this, I am planing to dump the latest offset
into graphite for each partition and look at derivative over time.
Thanks,
Bhavesh
Re: Get Latest Offset for Specific Topic for All Partition
Posted by Gwen Shapira <gs...@cloudera.com>.
Yes, its the size of the log in bytes. We can improve on documentation a
bit :)
Metric is defined in Log.scala:
newGauge("Size",
new Gauge[Long] {
def value = size
},
tags)
so the value is "size", and:
/**
* The size of the log in bytes
*/
def size: Long = logSegments.map(_.size).sum
So here we go :)
Gwen
On Mon, Feb 9, 2015 at 11:08 PM, Bhavesh Mistry <mi...@gmail.com>
wrote:
> Hi Gwen,
>
> This JMX stats is good for calculate injection rate per partition. I do
> not have to depend on ZK to figuring out who is leader what is latest
> offset.
>
> One quick question, what is Size # ? is it # of bytes particular
> partition has on disk ? Unfortunately, MBean description is very limited
> and does not help much(it is "Information on the management interface of
> the MBean"). It is gauge that is all I can tell .
>
> 1189855393 LogEndOffset
> 1165330350 Size
> 1176813232 LogStartOffset
>
> Thanks for your help !!
>
> Thanks,
>
> Bhaevsh
>
> Thanks,
> Bhavesh
>
> On Thu, Feb 5, 2015 at 12:55 PM, Gwen Shapira <gs...@cloudera.com>
> wrote:
>
> > You can use the metrics Kafka publishes. I think the relevant metrics
> are:
> > Log.LogEndOffset
> > Log.LogStartOffset
> > Log.size
> >
> >
> > Gwen
> >
> > On Thu, Feb 5, 2015 at 11:54 AM, Bhavesh Mistry
> > <mi...@gmail.com> wrote:
> > > HI All,
> > >
> > > I just need to get the latest offset # for topic (not for consumer
> > group).
> > > Which API to get this info ?
> > >
> > > My use case is to analyze the data injection rate to each of partition
> > is
> > > uniform or not (close). For this, I am planing to dump the latest
> offset
> > > into graphite for each partition and look at derivative over time.
> > >
> > > Thanks,
> > >
> > > Bhavesh
> >
>
Re: Get Latest Offset for Specific Topic for All Partition
Posted by Bhavesh Mistry <mi...@gmail.com>.
Hi Gwen,
This JMX stats is good for calculate injection rate per partition. I do
not have to depend on ZK to figuring out who is leader what is latest
offset.
One quick question, what is Size # ? is it # of bytes particular
partition has on disk ? Unfortunately, MBean description is very limited
and does not help much(it is "Information on the management interface of
the MBean"). It is gauge that is all I can tell .
1189855393 LogEndOffset
1165330350 Size
1176813232 LogStartOffset
Thanks for your help !!
Thanks,
Bhaevsh
Thanks,
Bhavesh
On Thu, Feb 5, 2015 at 12:55 PM, Gwen Shapira <gs...@cloudera.com> wrote:
> You can use the metrics Kafka publishes. I think the relevant metrics are:
> Log.LogEndOffset
> Log.LogStartOffset
> Log.size
>
>
> Gwen
>
> On Thu, Feb 5, 2015 at 11:54 AM, Bhavesh Mistry
> <mi...@gmail.com> wrote:
> > HI All,
> >
> > I just need to get the latest offset # for topic (not for consumer
> group).
> > Which API to get this info ?
> >
> > My use case is to analyze the data injection rate to each of partition
> is
> > uniform or not (close). For this, I am planing to dump the latest offset
> > into graphite for each partition and look at derivative over time.
> >
> > Thanks,
> >
> > Bhavesh
>
Re: Get Latest Offset for Specific Topic for All Partition
Posted by Gwen Shapira <gs...@cloudera.com>.
You can use the metrics Kafka publishes. I think the relevant metrics are:
Log.LogEndOffset
Log.LogStartOffset
Log.size
Gwen
On Thu, Feb 5, 2015 at 11:54 AM, Bhavesh Mistry
<mi...@gmail.com> wrote:
> HI All,
>
> I just need to get the latest offset # for topic (not for consumer group).
> Which API to get this info ?
>
> My use case is to analyze the data injection rate to each of partition is
> uniform or not (close). For this, I am planing to dump the latest offset
> into graphite for each partition and look at derivative over time.
>
> Thanks,
>
> Bhavesh
Re: Get Latest Offset for Specific Topic for All Partition
Posted by Joel Koshy <jj...@gmail.com>.
https://cwiki.apache.org/confluence/display/KAFKA/FAQ#FAQ-HowdoIaccuratelygetoffsetsofmessagesforacertaintimestampusingOffsetRequest?
However, you will need to issue a TopicMetadataRequest first to
discover the leaders for all the partitions and then issue the offset
request.
On Thu, Feb 05, 2015 at 11:54:00AM -0800, Bhavesh Mistry wrote:
> HI All,
>
> I just need to get the latest offset # for topic (not for consumer group).
> Which API to get this info ?
>
> My use case is to analyze the data injection rate to each of partition is
> uniform or not (close). For this, I am planing to dump the latest offset
> into graphite for each partition and look at derivative over time.
>
> Thanks,
>
> Bhavesh