[jira] [Created] (FLINK-29270) Wrong metrics kafka producer (FlinkKafkaProducer, KafkaSink) in EXACTLY_ONCE

["Muhammad Hilmi Al Fatih (Jira)" <ji...@apache.org>]

Muhammad Hilmi Al Fatih created FLINK-29270:
-----------------------------------------------

             Summary: Wrong metrics kafka producer (FlinkKafkaProducer, KafkaSink) in EXACTLY_ONCE
                 Key: FLINK-29270
                 URL: https://issues.apache.org/jira/browse/FLINK-29270
             Project: Flink
          Issue Type: Bug
          Components: Connectors / Kafka, Runtime / Metrics
    Affects Versions: 1.14.4
            Reporter: Muhammad Hilmi Al Fatih
         Attachments: image-2022-09-13-11-41-51-759.png, image-2022-09-13-11-42-45-297.png, image-2022-09-13-11-43-38-226.png, image-2022-09-13-11-43-50-441.png

I did a very simple kafka-to-kafka pipeline with flink 1.14.4 with checkpoint.

I did 4 tests to confirm: for each old API (FlinkKafkaConsumer&FlinkKafkaProducer) and new API (KafkaSource&KafkaSink), I run AT_LEAST_ONCE and EXACLTY_ONCE.

However, the metrics shown by producer is not correct when doing EXACTLY_ONCE semantics in both APIs.

 

To understand the situation, I also check the incoming rate metrics for both source and sink topics. So here are the summary:
h3. Old API
h4. At Least Once

!image-2022-09-13-11-41-51-759.png|width=616,height=190!
h4. Exactly Once

!image-2022-09-13-11-42-45-297.png!
h3. New API (KafkaSource&KafkaSink)
h4. At Least Once

!image-2022-09-13-11-43-38-226.png!
h4. Exactly Once

!image-2022-09-13-11-43-50-441.png!



--
This message was sent by Atlassian Jira
(v8.20.10#820010)