You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Sining Ma <si...@aol.com> on 2013/05/16 03:05:09 UTC
API to to query messages amount under one topic
Hi,
Is there any APIs in kafka that I can use to query how many messages there are under one topic?
This topic already exist in Kafka server, and producer is sending messages to one topic in kafka server
-- Regards
Sining Ma
RE: API to to query messages amount under one topic
Posted by Rob Withers <re...@gmail.com>.
+ 1!
A call through the Consumer Group, queuedMessageCount(topic,
partitionNumber), could query zookeeper (and add the local fetched queue) to
return this interesting piece of information. That is if we cannot already
do this :)
Thanks,
rob
> -----Original Message-----
> From: Sining Ma [mailto:siningma@aol.com]
> Sent: Wednesday, May 15, 2013 7:05 PM
> To: users@kafka.apache.org
> Subject: API to to query messages amount under one topic
>
> Hi,
> Is there any APIs in kafka that I can use to query how many messages there
> are under one topic?
> This topic already exist in Kafka server, and producer is sending messages
to
> one topic in kafka server
>
>
> -- Regards
> Sining Ma
>
Re: API to to query messages amount under one topic
Posted by Jun Rao <ju...@gmail.com>.
Yes, that's probably what you are looking for. It tells you the # of
unconsumed messages per partition, for a particular consumer.
Thanks,
Jun
On Fri, May 17, 2013 at 4:52 PM, Rob Withers <re...@gmail.com> wrote:
> Found out about the following:
>
> JMX -> “kafka.server” -> (various FeatcherThread)-ConsumerLag
>
> this sounds like the latency, in time. is it so?
>
> thanks,
> rob
>
> On May 17, 2013, at 11:49 AM, "Withers, Robert" <Ro...@dish.com>
> wrote:
>
> > Could you add some JMX stats for us, then?
> >
> > - Queue length, by group offset vs lastOffset
> > - latency between produce and consume, again by group
> >
> > Thanks,
> >
> >
> > Rob Withers
> > Staff Analyst/Developer
> > o: (720) 514-8963
> > c: (571) 262-1873
> >
> >
> >
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Friday, May 17, 2013 8:52 AM
> > To: users@kafka.apache.org
> > Subject: Re: API to to query messages amount under one topic
> >
> > For monitoring, we have jmxs on the broker for both the message and the
> byte rate.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 16, 2013 at 10:04 PM, Rob Withers <re...@gmail.com>
> wrote:
> >
> >> Immediately monitoring. Later, possible thresholding to evoke a
> >> reconfiguration of the number of partitions into a new topic and
> >> migrate message index/logs and redirect pubs/subs to the new topic,
> >> during a traffic spike.
> >>
> >>> -----Original Message-----
> >>> From: Jun Rao [mailto:junrao@gmail.com]
> >>> Sent: Thursday, May 16, 2013 10:00 PM
> >>> To: users@kafka.apache.org
> >>> Subject: Re: API to to query messages amount under one topic
> >>>
> >>> What do you plan to use this information for?
> >>>
> >>> Thanks,
> >>>
> >>> Jun
> >>>
> >>>
> >>> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> >>> <Ro...@dish.com>wrote:
> >>>
> >>>> Any possibility for a queuedMessageCount(topic, partitionNumber)
> >>> protocol?
> >>>>
> >>>> thanks,
> >>>> rob
> >>>> ________________________________________
> >>>> From: Jun Rao [junrao@gmail.com]
> >>>> Sent: Wednesday, May 15, 2013 10:59 PM
> >>>> To: users@kafka.apache.org
> >>>> Subject: Re: API to to query messages amount under one topic
> >>>>
> >>>> In 0.8, you can get the earliest and the latest offset using the
> >>>> getOffsetBefore api. The difference btw the two gives the number
> >>>> of messages on the broker.
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>>
> >>>> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> >>>>
> >>>>> Hi,
> >>>>> Is there any APIs in kafka that I can use to query how many
> >>>>> messages
> >>>> there
> >>>>> are under one topic?
> >>>>> This topic already exist in Kafka server, and producer is
> >>>>> sending
> >>>> messages
> >>>>> to one topic in kafka server
> >>>>>
> >>>>>
> >>>>> -- Regards
> >>>>> Sining Ma
> >>>>>
> >>>>>
> >>>>>
> >>>>
> >>
> >>
>
>
Re: API to to query messages amount under one topic
Posted by Rob Withers <re...@gmail.com>.
Found out about the following:
JMX -> “kafka.server” -> (various FeatcherThread)-ConsumerLag
this sounds like the latency, in time. is it so?
thanks,
rob
On May 17, 2013, at 11:49 AM, "Withers, Robert" <Ro...@dish.com> wrote:
> Could you add some JMX stats for us, then?
>
> - Queue length, by group offset vs lastOffset
> - latency between produce and consume, again by group
>
> Thanks,
>
>
> Rob Withers
> Staff Analyst/Developer
> o: (720) 514-8963
> c: (571) 262-1873
>
>
>
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Friday, May 17, 2013 8:52 AM
> To: users@kafka.apache.org
> Subject: Re: API to to query messages amount under one topic
>
> For monitoring, we have jmxs on the broker for both the message and the byte rate.
>
> Thanks,
>
> Jun
>
>
> On Thu, May 16, 2013 at 10:04 PM, Rob Withers <re...@gmail.com> wrote:
>
>> Immediately monitoring. Later, possible thresholding to evoke a
>> reconfiguration of the number of partitions into a new topic and
>> migrate message index/logs and redirect pubs/subs to the new topic,
>> during a traffic spike.
>>
>>> -----Original Message-----
>>> From: Jun Rao [mailto:junrao@gmail.com]
>>> Sent: Thursday, May 16, 2013 10:00 PM
>>> To: users@kafka.apache.org
>>> Subject: Re: API to to query messages amount under one topic
>>>
>>> What do you plan to use this information for?
>>>
>>> Thanks,
>>>
>>> Jun
>>>
>>>
>>> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
>>> <Ro...@dish.com>wrote:
>>>
>>>> Any possibility for a queuedMessageCount(topic, partitionNumber)
>>> protocol?
>>>>
>>>> thanks,
>>>> rob
>>>> ________________________________________
>>>> From: Jun Rao [junrao@gmail.com]
>>>> Sent: Wednesday, May 15, 2013 10:59 PM
>>>> To: users@kafka.apache.org
>>>> Subject: Re: API to to query messages amount under one topic
>>>>
>>>> In 0.8, you can get the earliest and the latest offset using the
>>>> getOffsetBefore api. The difference btw the two gives the number
>>>> of messages on the broker.
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>>
>>>> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
>>>>
>>>>> Hi,
>>>>> Is there any APIs in kafka that I can use to query how many
>>>>> messages
>>>> there
>>>>> are under one topic?
>>>>> This topic already exist in Kafka server, and producer is
>>>>> sending
>>>> messages
>>>>> to one topic in kafka server
>>>>>
>>>>>
>>>>> -- Regards
>>>>> Sining Ma
>>>>>
>>>>>
>>>>>
>>>>
>>
>>
RE: API to to query messages amount under one topic
Posted by "Withers, Robert" <Ro...@dish.com>.
Could you add some JMX stats for us, then?
- Queue length, by group offset vs lastOffset
- latency between produce and consume, again by group
Thanks,
Rob Withers
Staff Analyst/Developer
o: (720) 514-8963
c: (571) 262-1873
-----Original Message-----
From: Jun Rao [mailto:junrao@gmail.com]
Sent: Friday, May 17, 2013 8:52 AM
To: users@kafka.apache.org
Subject: Re: API to to query messages amount under one topic
For monitoring, we have jmxs on the broker for both the message and the byte rate.
Thanks,
Jun
On Thu, May 16, 2013 at 10:04 PM, Rob Withers <re...@gmail.com> wrote:
> Immediately monitoring. Later, possible thresholding to evoke a
> reconfiguration of the number of partitions into a new topic and
> migrate message index/logs and redirect pubs/subs to the new topic,
> during a traffic spike.
>
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, May 16, 2013 10:00 PM
> > To: users@kafka.apache.org
> > Subject: Re: API to to query messages amount under one topic
> >
> > What do you plan to use this information for?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> > <Ro...@dish.com>wrote:
> >
> > > Any possibility for a queuedMessageCount(topic, partitionNumber)
> > protocol?
> > >
> > > thanks,
> > > rob
> > > ________________________________________
> > > From: Jun Rao [junrao@gmail.com]
> > > Sent: Wednesday, May 15, 2013 10:59 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: API to to query messages amount under one topic
> > >
> > > In 0.8, you can get the earliest and the latest offset using the
> > > getOffsetBefore api. The difference btw the two gives the number
> > > of messages on the broker.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> > >
> > > > Hi,
> > > > Is there any APIs in kafka that I can use to query how many
> > > > messages
> > > there
> > > > are under one topic?
> > > > This topic already exist in Kafka server, and producer is
> > > > sending
> > > messages
> > > > to one topic in kafka server
> > > >
> > > >
> > > > -- Regards
> > > > Sining Ma
> > > >
> > > >
> > > >
> > >
>
>
Re: API to to query messages amount under one topic
Posted by Jun Rao <ju...@gmail.com>.
For monitoring, we have jmxs on the broker for both the message and the
byte rate.
Thanks,
Jun
On Thu, May 16, 2013 at 10:04 PM, Rob Withers <re...@gmail.com> wrote:
> Immediately monitoring. Later, possible thresholding to evoke a
> reconfiguration of the number of partitions into a new topic and migrate
> message index/logs and redirect pubs/subs to the new topic, during a
> traffic
> spike.
>
> > -----Original Message-----
> > From: Jun Rao [mailto:junrao@gmail.com]
> > Sent: Thursday, May 16, 2013 10:00 PM
> > To: users@kafka.apache.org
> > Subject: Re: API to to query messages amount under one topic
> >
> > What do you plan to use this information for?
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> > <Ro...@dish.com>wrote:
> >
> > > Any possibility for a queuedMessageCount(topic, partitionNumber)
> > protocol?
> > >
> > > thanks,
> > > rob
> > > ________________________________________
> > > From: Jun Rao [junrao@gmail.com]
> > > Sent: Wednesday, May 15, 2013 10:59 PM
> > > To: users@kafka.apache.org
> > > Subject: Re: API to to query messages amount under one topic
> > >
> > > In 0.8, you can get the earliest and the latest offset using the
> > > getOffsetBefore api. The difference btw the two gives the number of
> > > messages on the broker.
> > >
> > > Thanks,
> > >
> > > Jun
> > >
> > >
> > > On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> > >
> > > > Hi,
> > > > Is there any APIs in kafka that I can use to query how many messages
> > > there
> > > > are under one topic?
> > > > This topic already exist in Kafka server, and producer is sending
> > > messages
> > > > to one topic in kafka server
> > > >
> > > >
> > > > -- Regards
> > > > Sining Ma
> > > >
> > > >
> > > >
> > >
>
>
RE: API to to query messages amount under one topic
Posted by Rob Withers <re...@gmail.com>.
Immediately monitoring. Later, possible thresholding to evoke a
reconfiguration of the number of partitions into a new topic and migrate
message index/logs and redirect pubs/subs to the new topic, during a traffic
spike.
> -----Original Message-----
> From: Jun Rao [mailto:junrao@gmail.com]
> Sent: Thursday, May 16, 2013 10:00 PM
> To: users@kafka.apache.org
> Subject: Re: API to to query messages amount under one topic
>
> What do you plan to use this information for?
>
> Thanks,
>
> Jun
>
>
> On Thu, May 16, 2013 at 5:57 AM, Withers, Robert
> <Ro...@dish.com>wrote:
>
> > Any possibility for a queuedMessageCount(topic, partitionNumber)
> protocol?
> >
> > thanks,
> > rob
> > ________________________________________
> > From: Jun Rao [junrao@gmail.com]
> > Sent: Wednesday, May 15, 2013 10:59 PM
> > To: users@kafka.apache.org
> > Subject: Re: API to to query messages amount under one topic
> >
> > In 0.8, you can get the earliest and the latest offset using the
> > getOffsetBefore api. The difference btw the two gives the number of
> > messages on the broker.
> >
> > Thanks,
> >
> > Jun
> >
> >
> > On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> >
> > > Hi,
> > > Is there any APIs in kafka that I can use to query how many messages
> > there
> > > are under one topic?
> > > This topic already exist in Kafka server, and producer is sending
> > messages
> > > to one topic in kafka server
> > >
> > >
> > > -- Regards
> > > Sining Ma
> > >
> > >
> > >
> >
Re: API to to query messages amount under one topic
Posted by Jun Rao <ju...@gmail.com>.
What do you plan to use this information for?
Thanks,
Jun
On Thu, May 16, 2013 at 5:57 AM, Withers, Robert <Ro...@dish.com>wrote:
> Any possibility for a queuedMessageCount(topic, partitionNumber) protocol?
>
> thanks,
> rob
> ________________________________________
> From: Jun Rao [junrao@gmail.com]
> Sent: Wednesday, May 15, 2013 10:59 PM
> To: users@kafka.apache.org
> Subject: Re: API to to query messages amount under one topic
>
> In 0.8, you can get the earliest and the latest offset using the
> getOffsetBefore api. The difference btw the two gives the number of
> messages on the broker.
>
> Thanks,
>
> Jun
>
>
> On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
>
> > Hi,
> > Is there any APIs in kafka that I can use to query how many messages
> there
> > are under one topic?
> > This topic already exist in Kafka server, and producer is sending
> messages
> > to one topic in kafka server
> >
> >
> > -- Regards
> > Sining Ma
> >
> >
> >
>
RE: API to to query messages amount under one topic
Posted by "Withers, Robert" <Ro...@dish.com>.
Any possibility for a queuedMessageCount(topic, partitionNumber) protocol?
thanks,
rob
________________________________________
From: Jun Rao [junrao@gmail.com]
Sent: Wednesday, May 15, 2013 10:59 PM
To: users@kafka.apache.org
Subject: Re: API to to query messages amount under one topic
In 0.8, you can get the earliest and the latest offset using the
getOffsetBefore api. The difference btw the two gives the number of
messages on the broker.
Thanks,
Jun
On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> Hi,
> Is there any APIs in kafka that I can use to query how many messages there
> are under one topic?
> This topic already exist in Kafka server, and producer is sending messages
> to one topic in kafka server
>
>
> -- Regards
> Sining Ma
>
>
>
Re: API to to query messages amount under one topic
Posted by Jun Rao <ju...@gmail.com>.
In 0.8, you can get the earliest and the latest offset using the
getOffsetBefore api. The difference btw the two gives the number of
messages on the broker.
Thanks,
Jun
On Wed, May 15, 2013 at 6:05 PM, Sining Ma <si...@aol.com> wrote:
> Hi,
> Is there any APIs in kafka that I can use to query how many messages there
> are under one topic?
> This topic already exist in Kafka server, and producer is sending messages
> to one topic in kafka server
>
>
> -- Regards
> Sining Ma
>
>
>