You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Joseph Boyd <jo...@cbsinteractive.com> on 2012/08/01 19:10:23 UTC
Any way to know from kafka's JMX interface whether there are unread
messages on a topic
Hi there,
Is there any way to know from kafka's JMX interface whether there are
unread messages on a topic? I think the answer is 'no', based on my
read of documentation (esp. the 'high water mark'), and my own
experiments with producing test messages, looking at the stats, and
then consuming messages (both with/without committing messages
consumed).
I thought I'd double check on the users list anyhow, though, in case
I'm missing something. The KAFKA-260 work looks useful in this area
too.
Thanks for any help.
...joe
Re: Any way to know from kafka's JMX interface whether there are
unread messages on a topic
Posted by Jun Rao <ju...@gmail.com>.
Corbin,
The JMX beans and their methods can be found at
https://cwiki.apache.org/confluence/display/KAFKA/Operations#Operations-Monitoring
Thanks,
Jun
On Sun, Aug 5, 2012 at 8:08 AM, Corbin Hoenes <co...@tynt.com> wrote:
> Along these lines is there a list of JMX beans and what values are exposed
> w/some description of the metric they communicate?
> I am trying to understand how the system is doing and don't know what I
> don't know yet :)
>
> On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:
>
> > Thanks! I was (wrongly) looking for this information on the kafka
> > process, and not on the consumer process. Once I looked there I found
> > kafka.ConsumerStats.
> >
> > (still not sure how I read dates wrong for the JIRA, but that happens)
> >
> >
> > ...joe
> >
> >
> > On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <ju...@gmail.com> wrote:
> >> Joe,
> >>
> >> It should be there. 0.7.1 was released last month and that jira was
> closed
> >> last year.
> >>
> >> Thanks,
> >>
> >> Jun
> >>
> >> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <
> joseph.boyd@cbsinteractive.com
> >>> wrote:
> >>
> >>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
> >>> seeing it.
> >>>
> >>> My read of JIRA / mailing list archives shows KAFKA-136 went in
> >>> shortly after the 0.7.1 release was rolled up.
> >>>
> >>>
> >>> Thanks for the pointer, though.
> >>>
> >>>
> >>> ...joe
> >>>
> >>>
> >>>
> >>>
> >>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <ju...@gmail.com> wrote:
> >>>> Joe,
> >>>>
> >>>> We have the following bean name and type registered for consumers.
> There
> >>> is
> >>>> also a CheckOffsetLag tool.
> >>>>
> >>>> kafka:type=kafka.ConsumerStats
> >>>> /**
> >>>> * JMX interface for monitoring consumer
> >>>> */
> >>>> trait ZookeeperConsumerConnectorMBean {
> >>>> def getPartOwnerStats: String
> >>>> def getConsumerGroup: String
> >>>> def getOffsetLag(topic: String, brokerId: Int, partitionId: Int):
> Long
> >>>> def getConsumedOffset(topic: String, brokerId: Int, partitionId:
> Int):
> >>>> Long
> >>>> def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
> >>> Long
> >>>> }
> >>>>
> >>>> Thanks,
> >>>>
> >>>> Jun
> >>>>
> >>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
> >>> joseph.boyd@cbsinteractive.com
> >>>>> wrote:
> >>>>
> >>>>> Hi there,
> >>>>>
> >>>>> Is there any way to know from kafka's JMX interface whether there are
> >>>>> unread messages on a topic? I think the answer is 'no', based on my
> >>>>> read of documentation (esp. the 'high water mark'), and my own
> >>>>> experiments with producing test messages, looking at the stats, and
> >>>>> then consuming messages (both with/without committing messages
> >>>>> consumed).
> >>>>>
> >>>>> I thought I'd double check on the users list anyhow, though, in case
> >>>>> I'm missing something. The KAFKA-260 work looks useful in this area
> >>>>> too.
> >>>>>
> >>>>> Thanks for any help.
> >>>>>
> >>>>>
> >>>>> ...joe
> >>>>>
> >>>
>
>
Re: Any way to know from kafka's JMX interface whether there are unread messages on a topic
Posted by Corbin Hoenes <co...@tynt.com>.
Along these lines is there a list of JMX beans and what values are exposed w/some description of the metric they communicate?
I am trying to understand how the system is doing and don't know what I don't know yet :)
On Aug 2, 2012, at 11:20 AM, Joseph Boyd wrote:
> Thanks! I was (wrongly) looking for this information on the kafka
> process, and not on the consumer process. Once I looked there I found
> kafka.ConsumerStats.
>
> (still not sure how I read dates wrong for the JIRA, but that happens)
>
>
> ...joe
>
>
> On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <ju...@gmail.com> wrote:
>> Joe,
>>
>> It should be there. 0.7.1 was released last month and that jira was closed
>> last year.
>>
>> Thanks,
>>
>> Jun
>>
>> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <joseph.boyd@cbsinteractive.com
>>> wrote:
>>
>>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>>> seeing it.
>>>
>>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>>> shortly after the 0.7.1 release was rolled up.
>>>
>>>
>>> Thanks for the pointer, though.
>>>
>>>
>>> ...joe
>>>
>>>
>>>
>>>
>>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <ju...@gmail.com> wrote:
>>>> Joe,
>>>>
>>>> We have the following bean name and type registered for consumers. There
>>> is
>>>> also a CheckOffsetLag tool.
>>>>
>>>> kafka:type=kafka.ConsumerStats
>>>> /**
>>>> * JMX interface for monitoring consumer
>>>> */
>>>> trait ZookeeperConsumerConnectorMBean {
>>>> def getPartOwnerStats: String
>>>> def getConsumerGroup: String
>>>> def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>>>> def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>>>> Long
>>>> def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>>> Long
>>>> }
>>>>
>>>> Thanks,
>>>>
>>>> Jun
>>>>
>>>> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>>> joseph.boyd@cbsinteractive.com
>>>>> wrote:
>>>>
>>>>> Hi there,
>>>>>
>>>>> Is there any way to know from kafka's JMX interface whether there are
>>>>> unread messages on a topic? I think the answer is 'no', based on my
>>>>> read of documentation (esp. the 'high water mark'), and my own
>>>>> experiments with producing test messages, looking at the stats, and
>>>>> then consuming messages (both with/without committing messages
>>>>> consumed).
>>>>>
>>>>> I thought I'd double check on the users list anyhow, though, in case
>>>>> I'm missing something. The KAFKA-260 work looks useful in this area
>>>>> too.
>>>>>
>>>>> Thanks for any help.
>>>>>
>>>>>
>>>>> ...joe
>>>>>
>>>
Re: Any way to know from kafka's JMX interface whether there are
unread messages on a topic
Posted by Joseph Boyd <jo...@cbsinteractive.com>.
Thanks! I was (wrongly) looking for this information on the kafka
process, and not on the consumer process. Once I looked there I found
kafka.ConsumerStats.
(still not sure how I read dates wrong for the JIRA, but that happens)
...joe
On Wed, Aug 1, 2012 at 10:22 PM, Jun Rao <ju...@gmail.com> wrote:
> Joe,
>
> It should be there. 0.7.1 was released last month and that jira was closed
> last year.
>
> Thanks,
>
> Jun
>
> On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <joseph.boyd@cbsinteractive.com
>> wrote:
>
>> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
>> seeing it.
>>
>> My read of JIRA / mailing list archives shows KAFKA-136 went in
>> shortly after the 0.7.1 release was rolled up.
>>
>>
>> Thanks for the pointer, though.
>>
>>
>> ...joe
>>
>>
>>
>>
>> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <ju...@gmail.com> wrote:
>> > Joe,
>> >
>> > We have the following bean name and type registered for consumers. There
>> is
>> > also a CheckOffsetLag tool.
>> >
>> > kafka:type=kafka.ConsumerStats
>> > /**
>> > * JMX interface for monitoring consumer
>> > */
>> > trait ZookeeperConsumerConnectorMBean {
>> > def getPartOwnerStats: String
>> > def getConsumerGroup: String
>> > def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
>> > def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
>> > Long
>> > def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
>> Long
>> > }
>> >
>> > Thanks,
>> >
>> > Jun
>> >
>> > On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
>> joseph.boyd@cbsinteractive.com
>> >> wrote:
>> >
>> >> Hi there,
>> >>
>> >> Is there any way to know from kafka's JMX interface whether there are
>> >> unread messages on a topic? I think the answer is 'no', based on my
>> >> read of documentation (esp. the 'high water mark'), and my own
>> >> experiments with producing test messages, looking at the stats, and
>> >> then consuming messages (both with/without committing messages
>> >> consumed).
>> >>
>> >> I thought I'd double check on the users list anyhow, though, in case
>> >> I'm missing something. The KAFKA-260 work looks useful in this area
>> >> too.
>> >>
>> >> Thanks for any help.
>> >>
>> >>
>> >> ...joe
>> >>
>>
Re: Any way to know from kafka's JMX interface whether there are
unread messages on a topic
Posted by Jun Rao <ju...@gmail.com>.
Joe,
It should be there. 0.7.1 was released last month and that jira was closed
last year.
Thanks,
Jun
On Wed, Aug 1, 2012 at 10:51 AM, Joseph Boyd <joseph.boyd@cbsinteractive.com
> wrote:
> It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
> seeing it.
>
> My read of JIRA / mailing list archives shows KAFKA-136 went in
> shortly after the 0.7.1 release was rolled up.
>
>
> Thanks for the pointer, though.
>
>
> ...joe
>
>
>
>
> On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <ju...@gmail.com> wrote:
> > Joe,
> >
> > We have the following bean name and type registered for consumers. There
> is
> > also a CheckOffsetLag tool.
> >
> > kafka:type=kafka.ConsumerStats
> > /**
> > * JMX interface for monitoring consumer
> > */
> > trait ZookeeperConsumerConnectorMBean {
> > def getPartOwnerStats: String
> > def getConsumerGroup: String
> > def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
> > def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
> > Long
> > def getLatestOffset(topic: String, brokerId: Int, partitionId: Int):
> Long
> > }
> >
> > Thanks,
> >
> > Jun
> >
> > On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <
> joseph.boyd@cbsinteractive.com
> >> wrote:
> >
> >> Hi there,
> >>
> >> Is there any way to know from kafka's JMX interface whether there are
> >> unread messages on a topic? I think the answer is 'no', based on my
> >> read of documentation (esp. the 'high water mark'), and my own
> >> experiments with producing test messages, looking at the stats, and
> >> then consuming messages (both with/without committing messages
> >> consumed).
> >>
> >> I thought I'd double check on the users list anyhow, though, in case
> >> I'm missing something. The KAFKA-260 work looks useful in this area
> >> too.
> >>
> >> Thanks for any help.
> >>
> >>
> >> ...joe
> >>
>
Re: Any way to know from kafka's JMX interface whether there are
unread messages on a topic
Posted by Joseph Boyd <jo...@cbsinteractive.com>.
It looks like kafka.ConsumerStats isn't in 0.7.1, which is why I'm not
seeing it.
My read of JIRA / mailing list archives shows KAFKA-136 went in
shortly after the 0.7.1 release was rolled up.
Thanks for the pointer, though.
...joe
On Wed, Aug 1, 2012 at 10:19 AM, Jun Rao <ju...@gmail.com> wrote:
> Joe,
>
> We have the following bean name and type registered for consumers. There is
> also a CheckOffsetLag tool.
>
> kafka:type=kafka.ConsumerStats
> /**
> * JMX interface for monitoring consumer
> */
> trait ZookeeperConsumerConnectorMBean {
> def getPartOwnerStats: String
> def getConsumerGroup: String
> def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
> def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
> Long
> def getLatestOffset(topic: String, brokerId: Int, partitionId: Int): Long
> }
>
> Thanks,
>
> Jun
>
> On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <joseph.boyd@cbsinteractive.com
>> wrote:
>
>> Hi there,
>>
>> Is there any way to know from kafka's JMX interface whether there are
>> unread messages on a topic? I think the answer is 'no', based on my
>> read of documentation (esp. the 'high water mark'), and my own
>> experiments with producing test messages, looking at the stats, and
>> then consuming messages (both with/without committing messages
>> consumed).
>>
>> I thought I'd double check on the users list anyhow, though, in case
>> I'm missing something. The KAFKA-260 work looks useful in this area
>> too.
>>
>> Thanks for any help.
>>
>>
>> ...joe
>>
Re: Any way to know from kafka's JMX interface whether there are
unread messages on a topic
Posted by Jun Rao <ju...@gmail.com>.
Joe,
We have the following bean name and type registered for consumers. There is
also a CheckOffsetLag tool.
kafka:type=kafka.ConsumerStats
/**
* JMX interface for monitoring consumer
*/
trait ZookeeperConsumerConnectorMBean {
def getPartOwnerStats: String
def getConsumerGroup: String
def getOffsetLag(topic: String, brokerId: Int, partitionId: Int): Long
def getConsumedOffset(topic: String, brokerId: Int, partitionId: Int):
Long
def getLatestOffset(topic: String, brokerId: Int, partitionId: Int): Long
}
Thanks,
Jun
On Wed, Aug 1, 2012 at 10:10 AM, Joseph Boyd <joseph.boyd@cbsinteractive.com
> wrote:
> Hi there,
>
> Is there any way to know from kafka's JMX interface whether there are
> unread messages on a topic? I think the answer is 'no', based on my
> read of documentation (esp. the 'high water mark'), and my own
> experiments with producing test messages, looking at the stats, and
> then consuming messages (both with/without committing messages
> consumed).
>
> I thought I'd double check on the users list anyhow, though, in case
> I'm missing something. The KAFKA-260 work looks useful in this area
> too.
>
> Thanks for any help.
>
>
> ...joe
>