Kafka metrics to calculate number of messages in a topic

[Dhirendra Singh <dh...@gmail.com>]

Hi All,
I have a requirement to display the total number of messages in a topic in
grafana dashboard.
I am looking at the metrics exposed by kafka broker and came across the
following metrics.
kafka_log_log_logendoffset
kafka_log_log_logstartoffset

My understanding is that if I take the difference of
kafka_log_log_logendoffset and kafka_log_log_logstartoffset it should
result in the number of messages present in the topic.
Is my understanding correct ?

Thanks,
Dhirendra.

Re: Kafka metrics to calculate number of messages in a topic

[Eric Azama <ea...@gmail.com>]

That will get you a good approximation, but it's not guaranteed to be
completely accurate. Offsets in Kafka are not guaranteed to be continuous.

For topics with log compaction enabled, the removed records will leave
(potentially very large) holes in the offsets.

Even for topics without log compaction, it's possible for there to be small
holes in the offsets. Transaction markers for Exactly Once processing will
also use offsets.

On Mon, Aug 9, 2021 at 11:28 PM Dhirendra Singh <dh...@gmail.com>
wrote:

> Hi All,
> I have a requirement to display the total number of messages in a topic in
> grafana dashboard.
> I am looking at the metrics exposed by kafka broker and came across the
> following metrics.
> kafka_log_log_logendoffset
> kafka_log_log_logstartoffset
>
> My understanding is that if I take the difference of
> kafka_log_log_logendoffset and kafka_log_log_logstartoffset it should
> result in the number of messages present in the topic.
> Is my understanding correct ?
>
> Thanks,
> Dhirendra.
>