Consumer Group Lag Reporting

[Kyle Banker <ky...@gmail.com>]

What is the best practice for reporting the lag on individual consumer
groups (e.g., to Graphite)?

A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to
indicate that parsing the output of the consumer offset checker tool and
reporting that independently is what folks do. Is there a better way?

Are there any plans for the Kafka brokers themselves to do this reporting
(e.g., the master for each partition reports lag for all of its consumer
groups)?

Many thanks in advance.

Re: Consumer Group Lag Reporting

[Kyle Banker <ky...@gmail.com>]

Thanks, Otis. I actually already have a reporting and alerting
infrastructure.

I mainly wanted to confirm that parsing the output of the offset checker is
the recommended practice for reporting consumer group offsets. Is this the
case? If so, I wanted to find out if any work is under way to make this
more streamlined.

On Mon, Apr 6, 2015 at 10:05 PM, Otis Gospodnetic <
otis.gospodnetic@gmail.com> wrote:

> Hi,
>
> We track our Consumer Lag and all other Kafka metrics with SPM
> <http://sematext.com/spm> whose agent collects this info from Kafka
> Consumers.  Here's a chart of that:
> https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart
> doesn't show all filtering option, but you can filter lag by host, client
> id, topic, and partition.
>
> Otis
> --
> Monitoring * Alerting * Anomaly Detection * Centralized Log Management
> Solr & Elasticsearch Support * http://sematext.com/
>
>
> On Mon, Apr 6, 2015 at 6:14 PM, Kyle Banker <ky...@gmail.com> wrote:
>
> > What is the best practice for reporting the lag on individual consumer
> > groups (e.g., to Graphite)?
> >
> > A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to
> > indicate that parsing the output of the consumer offset checker tool and
> > reporting that independently is what folks do. Is there a better way?
> >
> > Are there any plans for the Kafka brokers themselves to do this reporting
> > (e.g., the master for each partition reports lag for all of its consumer
> > groups)?
> >
> > Many thanks in advance.
> >
>

Re: Consumer Group Lag Reporting

[Otis Gospodnetic <ot...@gmail.com>]

Hi,

We track our Consumer Lag and all other Kafka metrics with SPM
<http://sematext.com/spm> whose agent collects this info from Kafka
Consumers.  Here's a chart of that:
https://apps.sematext.com/spm-reports/s/mcBHvsXwgj -- this public chart
doesn't show all filtering option, but you can filter lag by host, client
id, topic, and partition.

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Mon, Apr 6, 2015 at 6:14 PM, Kyle Banker <ky...@gmail.com> wrote:

> What is the best practice for reporting the lag on individual consumer
> groups (e.g., to Graphite)?
>
> A recent form post (http://search-hadoop.com/m/4TaT4x9qWm1) seems to
> indicate that parsing the output of the consumer offset checker tool and
> reporting that independently is what folks do. Is there a better way?
>
> Are there any plans for the Kafka brokers themselves to do this reporting
> (e.g., the master for each partition reports lag for all of its consumer
> groups)?
>
> Many thanks in advance.
>