You are viewing a plain text version of this content. The canonical link for it is here.

Posted to issues@flink.apache.org by "zlzhang0122 (Jira)" <ji...@apache.org> on 2021/10/14 08:01:00 UTC

[jira] [Created] (FLINK-24542) Expose the freshness metrics for kafka connector

zlzhang0122 created FLINK-24542:
-----------------------------------

             Summary: Expose the freshness metrics for kafka connector
                 Key: FLINK-24542
                 URL: https://issues.apache.org/jira/browse/FLINK-24542
             Project: Flink
          Issue Type: Improvement
          Components: Connectors / Kafka
    Affects Versions: 1.13.1, 1.14.0, 1.12.2
            Reporter: zlzhang0122
             Fix For: 1.15.0


When we start a flink job to consume apache kafka, we usually use offsetLag, which can be calulated by current-offsets minus committed-offsets, but sometimes the offsetLag is hard to understand, we can hardly to judge wether the value is normal or not. Kafka have proposed a new metric: freshness(see [a guide to kafka consumer freshness|[https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat&from=timeline&isappinstalled=0])|https://www.jesseyates.com/2019/11/04/kafka-consumer-freshness-a-guide.html?trk=article_share_wechat&from=timeline&isappinstalled=0]),].

So we can also expose the freshness metric for kafka connector to improve the user experience.From this freshness metric, user can easily know wether the kafka message is backlog and need to deal with it.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)