How to monitor lag when "kafka" is used as offset.storage?

[shahab <sh...@gmail.com>]

Hi,

I wonder how we can monitor lag (difference between consumer offset and log
) when "kafka" is set as offset.storage?  because the "kafka-run-class.sh
kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper is
used as storage manager.

best,
/Shahab

Re: How to monitor lag when "kafka" is used as offset.storage?

[Todd Palino <tp...@gmail.com>]

You can use the emailer config in Burrow to send alerts directly (it will
monitor specific groups and send emails out when there is a problem). If
you need something more complex than that, I think the best practice is
always to send the output into an general alert/notification system.

-Todd

On Wednesday, September 2, 2015, shahab <sh...@gmail.com> wrote:

> Thanks Noah. I installed Burrow and played with it a little bit. It seems
> as you pointed out I need to implement the alerting system myself. Do you
> know any other Kafka tools that can give alerts?
>
> best,
> /Shahab
>
> On Wed, Sep 2, 2015 at 1:44 PM, noah <iamnoah@gmail.com <javascript:;>>
> wrote:
>
> > We use Burrow <https://github.com/linkedin/Burrow>. There are rest
> > endpoints you can use to get offsets and manually calculate lag, but if
> you
> > are focused on alerting, I'd use it's consumer statuses as they are a bit
> > smarter than a simple lag calculation.
> >
> > On Wed, Sep 2, 2015 at 4:08 AM shahab <shahab.mokari@gmail.com
> <javascript:;>> wrote:
> >
> > > Hi,
> > >
> > > I wonder how we can monitor lag (difference between consumer offset and
> > log
> > > ) when "kafka" is set as offset.storage?  because the
> "kafka-run-class.sh
> > > kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper
> is
> > > used as storage manager.
> > >
> > > best,
> > > /Shahab
> > >
> >
>

Re: How to monitor lag when "kafka" is used as offset.storage?

[Otis Gospodnetić <ot...@gmail.com>]

Hi Shahab - SPM for Kafka captures ~200 Kafka metrics IIRC and has built-in
alerting, anomaly detection, and a bunch of other features - see
http://sematext.com/spm/integrations/kafka-monitoring.html

Otis
--
Monitoring * Alerting * Anomaly Detection * Centralized Log Management
Solr & Elasticsearch Support * http://sematext.com/


On Thu, Sep 3, 2015 at 2:37 AM, shahab <sh...@gmail.com> wrote:

> Thanks Noah. I installed Burrow and played with it a little bit. It seems
> as you pointed out I need to implement the alerting system myself. Do you
> know any other Kafka tools that can give alerts?
>
> best,
> /Shahab
>
> On Wed, Sep 2, 2015 at 1:44 PM, noah <ia...@gmail.com> wrote:
>
> > We use Burrow <https://github.com/linkedin/Burrow>. There are rest
> > endpoints you can use to get offsets and manually calculate lag, but if
> you
> > are focused on alerting, I'd use it's consumer statuses as they are a bit
> > smarter than a simple lag calculation.
> >
> > On Wed, Sep 2, 2015 at 4:08 AM shahab <sh...@gmail.com> wrote:
> >
> > > Hi,
> > >
> > > I wonder how we can monitor lag (difference between consumer offset and
> > log
> > > ) when "kafka" is set as offset.storage?  because the
> "kafka-run-class.sh
> > > kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper
> is
> > > used as storage manager.
> > >
> > > best,
> > > /Shahab
> > >
> >
>

Re: How to monitor lag when "kafka" is used as offset.storage?

[shahab <sh...@gmail.com>]

Thanks Noah. I installed Burrow and played with it a little bit. It seems
as you pointed out I need to implement the alerting system myself. Do you
know any other Kafka tools that can give alerts?

best,
/Shahab

On Wed, Sep 2, 2015 at 1:44 PM, noah <ia...@gmail.com> wrote:

> We use Burrow <https://github.com/linkedin/Burrow>. There are rest
> endpoints you can use to get offsets and manually calculate lag, but if you
> are focused on alerting, I'd use it's consumer statuses as they are a bit
> smarter than a simple lag calculation.
>
> On Wed, Sep 2, 2015 at 4:08 AM shahab <sh...@gmail.com> wrote:
>
> > Hi,
> >
> > I wonder how we can monitor lag (difference between consumer offset and
> log
> > ) when "kafka" is set as offset.storage?  because the "kafka-run-class.sh
> > kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper is
> > used as storage manager.
> >
> > best,
> > /Shahab
> >
>

Re: How to monitor lag when "kafka" is used as offset.storage?

[noah <ia...@gmail.com>]

We use Burrow <https://github.com/linkedin/Burrow>. There are rest
endpoints you can use to get offsets and manually calculate lag, but if you
are focused on alerting, I'd use it's consumer statuses as they are a bit
smarter than a simple lag calculation.

On Wed, Sep 2, 2015 at 4:08 AM shahab <sh...@gmail.com> wrote:

> Hi,
>
> I wonder how we can monitor lag (difference between consumer offset and log
> ) when "kafka" is set as offset.storage?  because the "kafka-run-class.sh
> kafka.tools.ConsumerOffsetChecker ... " does work only when zookeeper is
> used as storage manager.
>
> best,
> /Shahab
>