[jira] [Commented] (KAFKA-13936) Invalid consumer lag when monitoring from a kafka streams application

["Matthias J. Sax (Jira)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-13936?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17542359#comment-17542359 ] 

Matthias J. Sax commented on KAFKA-13936:
-----------------------------------------

> we see in the kafka UI 

There is no "Kafka UI" – at least not as part of Apache Kafka. If you are using some other "external" UI, it's unclear how they compute/display the lag.

Two thories:
 * The don't sum the lag over all partitions but take the max over all partitions?
 * They compute the lag based on committed offsets to end offset difference – because offsets are committed only every 30 seconds by default, this computation does not match what the consumer reports, because the consumer uses "current offset" (not committed offset) to compute the lag

I think we need to close this ticket? I don't see any bug in Kafka (ie, consumer or Kafka Streams) that we would need to fix?

> Invalid consumer lag when monitoring from a kafka streams application
> ---------------------------------------------------------------------
>
>                 Key: KAFKA-13936
>                 URL: https://issues.apache.org/jira/browse/KAFKA-13936
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>            Reporter: Prashanth Joseph Babu
>            Priority: Major
>
> I have a kafka streams application and I'm trying to monitor the consumer lag via stream metrics.
> Here's some code snippet
> {code:java}
> metrics = streams.metrics();
>             lag = 0;
>             for (Metric m : metrics.values()) {
>                 tags = m.metricName().tags();
>                 if ( m.metricName().name().equals(MONITOR_CONSUMER_LAG) && tags.containsKey(MONTOR_TAG_TOPIC) && 
>                     tags.get(MONTOR_TAG_TOPIC).equals(inputTopic) ) {
>                     partitionLag = Float.valueOf(m.metricValue().toString()).floatValue();
>                     if ( !partitionLag.isNaN() ) {
>                         lag += partitionLag;
>                     }
>                 }
>             }
> {code}
> Here MONITOR_CONSUMER_LAG is {{{}records-lag-max{}}}.
> However these numbers dont match with the consumer lag we see in the kafka UI . is records-lag-max the right metric to track for a kafka streams application when the objective is to get consumer lag?



--
This message was sent by Atlassian Jira
(v8.20.7#820007)