Getting consumer offset lag in Kibana or grafana

["Vikas Bhatia -X (vikbhati - ARICENT TECHNOLOGIES MAURIITIUS LIMITED at Cisco)" <vi...@cisco.com>]

Hi,

I am running kafka version 0.8.2.

I have scenarios where there is single producer client and multiple consumer clients. All consumers consume from the same topic.
We differentiate consumers using the "consumerGroupId".

Now I have a requirement where I need offset value for each consumer.

I have tried using "kafka-consumer-offset-checker.sh" script and it gives me accurate results.
I have also explored kafka MBeans and it doesn't give me this lag data at the server level.

I am looking for best way to get the lag value and monitor it using kibana or grafana.

Please suggest what is the best approach for this.

Thanks and Regards
Vikas Bhatia

Re: Getting consumer offset lag in Kibana or grafana

[Gwen Shapira <gw...@confluent.io>]

As mentioned in our docs
(http://kafka.apache.org/082/documentation.html#monitoring) the 0.8.2
high level consumer has MBean called
kafka.consumer:type=ConsumerFetcherManager,name=MaxLag,clientId=([-.\w]+)

You should be able to use that.

On Wed, Sep 28, 2016 at 7:10 AM, Vikas Bhatia -X (vikbhati - ARICENT
TECHNOLOGIES MAURIITIUS LIMITED at Cisco) <vi...@cisco.com> wrote:
> Hi,
>
> I am running kafka version 0.8.2.
>
> I have scenarios where there is single producer client and multiple consumer clients. All consumers consume from the same topic.
> We differentiate consumers using the "consumerGroupId".
>
> Now I have a requirement where I need offset value for each consumer.
>
> I have tried using "kafka-consumer-offset-checker.sh" script and it gives me accurate results.
> I have also explored kafka MBeans and it doesn't give me this lag data at the server level.
>
> I am looking for best way to get the lag value and monitor it using kibana or grafana.
>
> Please suggest what is the best approach for this.
>
> Thanks and Regards
> Vikas Bhatia
>



-- 
Gwen Shapira
Product Manager | Confluent
650.450.2760 | @gwenshap
Follow us: Twitter | blog