Kafka metrics

[Eleanore Jin <el...@gmail.com>]

Hi community,

I just wonder what is the difference between the consumer lag reported by
Kafka client and the consumer lag reported by burrow?

Thanks a lot!
Eleanore

Re: Kafka metrics

["Matthias J. Sax" <mj...@apache.org>]

I am not 100% sure what Burrow does, but I would assume that it compares
committed offsets to end offsets (similar to
`bin/kafka-consumer-group.sh`). This is a "global" view over all
consumer in the group. Compare to the consumer metric, the might report
a higher lag as it relies on consumer commits.

The consumer lag metric reports a single client view (obviously a
consumer does not know anything about the lag of other consumers in the
group) and it's based on the current fetch offsets the consumer
maintains internally. Thus, the lag might be smaller if the offsets are
not committed yet.


-Matthias

On 5/11/20 5:23 PM, Eleanore Jin wrote:
> Hi community,
> 
> I just wonder what is the difference between the consumer lag reported by
> Kafka client and the consumer lag reported by burrow?
> 
> Thanks a lot!
> Eleanore
> 

Re: Kafka metrics

[Eleanore Jin <el...@gmail.com>]

Hi All,

Thanks a lot for the information and suggestion!

Eleanore

On Tue, May 12, 2020 at 3:24 PM Malcolm McFarland <mm...@cavulus.com>
wrote:

> These ideas are specific to Samza and ymmv in how they apply to other
> processing frameworks, but we use a couple of custom tools to keep tabs on
> processing lag:
>
> - one is a produce/consume timestamp comparison tool which utilizes writes
> a message production timestamps out to ZooKeeper on a per-partition basis;
> then in our stream processor, Samza, we then write out a consumption
> timestamp for the same partition to ZooKeeper, and we can use these
> differences (with some compensations for partition offset differences) to
> see what our processing lag is;
> - we also have a custom offset/checkpoint comparison tool which ingests
> Samza's checkpoint topic and compares it against the latest offsets on a
> per-partition basis to know how far behind each partition is in processing
> messages (this also doubles as a checkpoint-properties file generator which
> we can use to rebuild the checkpoint topic if it gets too large)
>
> These two tools have been invaluable in helping us monitor our Samza
> processing clusters.
>
> Cheers,
> Malcolm McFarland
> Cavulus
>
>
> This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
> unauthorized or improper disclosure, copying, distribution, or use of the
> contents of this message is prohibited. The information contained in this
> message is intended only for the personal and confidential use of the
> recipient(s) named above. If you have received this message in error,
> please notify the sender immediately and delete the original message.
>
>
> On Mon, May 11, 2020 at 5:23 PM Eleanore Jin <el...@gmail.com>
> wrote:
>
> > Hi community,
> >
> > I just wonder what is the difference between the consumer lag reported by
> > Kafka client and the consumer lag reported by burrow?
> >
> > Thanks a lot!
> > Eleanore
> >
>

Re: Kafka metrics

[Malcolm McFarland <mm...@cavulus.com>]

These ideas are specific to Samza and ymmv in how they apply to other
processing frameworks, but we use a couple of custom tools to keep tabs on
processing lag:

- one is a produce/consume timestamp comparison tool which utilizes writes
a message production timestamps out to ZooKeeper on a per-partition basis;
then in our stream processor, Samza, we then write out a consumption
timestamp for the same partition to ZooKeeper, and we can use these
differences (with some compensations for partition offset differences) to
see what our processing lag is;
- we also have a custom offset/checkpoint comparison tool which ingests
Samza's checkpoint topic and compares it against the latest offsets on a
per-partition basis to know how far behind each partition is in processing
messages (this also doubles as a checkpoint-properties file generator which
we can use to rebuild the checkpoint topic if it gets too large)

These two tools have been invaluable in helping us monitor our Samza
processing clusters.

Cheers,
Malcolm McFarland
Cavulus


This correspondence is from HealthPlanCRM, LLC, d/b/a Cavulus. Any
unauthorized or improper disclosure, copying, distribution, or use of the
contents of this message is prohibited. The information contained in this
message is intended only for the personal and confidential use of the
recipient(s) named above. If you have received this message in error,
please notify the sender immediately and delete the original message.


On Mon, May 11, 2020 at 5:23 PM Eleanore Jin <el...@gmail.com> wrote:

> Hi community,
>
> I just wonder what is the difference between the consumer lag reported by
> Kafka client and the consumer lag reported by burrow?
>
> Thanks a lot!
> Eleanore
>