How to know how many messages not yet processed ?

[Stone <st...@gmail.com>]

Our current messaging system is redis based (resque). So it's easy to know
how many messages got queued (not processed ) using LLEN to get the length
of a list (or using resque-web interface). For Kafka what's the recommended
way to get metrics such as # of unprocessed messages, how many messages got
by day or hour ? Is there any web based interface to view the metrics ?

Best Regards,
Stone

Re: How to know how many messages not yet processed ?

[Mark <st...@gmail.com>]

Stone,

We are in the same boat.. we are using Resque for our queueing and its 
invaluable to know this type of information. Unfortunately, I don't 
believe Kafka exposes this information. Perhaps another messaging system 
would better suit you... I belive RabbitMQ has an admin interface

On 12/30/11 5:59 AM, Stone wrote:
> Our current messaging system is redis based (resque). So it's easy to know
> how many messages got queued (not processed ) using LLEN to get the length
> of a list (or using resque-web interface). For Kafka what's the recommended
> way to get metrics such as # of unprocessed messages, how many messages got
> by day or hour ? Is there any web based interface to view the metrics ?
>
> Best Regards,
> Stone
>

Re: How to know how many messages not yet processed ?

[Jun Rao <ju...@gmail.com>]

Stone,

Currently, Kafka doesn't expose # of unconsumed messages. We do have a
monitoring tool ConsumerOffsetCheck that tells you the lag in bytes per
partition for a given consumer. There is also a jmx bean ConsumerTopicStat
that stores the # of consumed messages per topic for a consumer.

Thanks,

Jun

On Fri, Dec 30, 2011 at 5:59 AM, Stone <st...@gmail.com> wrote:

> Our current messaging system is redis based (resque). So it's easy to know
> how many messages got queued (not processed ) using LLEN to get the length
> of a list (or using resque-web interface). For Kafka what's the recommended
> way to get metrics such as # of unprocessed messages, how many messages got
> by day or hour ? Is there any web based interface to view the metrics ?
>
> Best Regards,
> Stone
>