[jira] [Updated] (FLINK-39892) Exactly-once kafka sink reports a negative numBytesOut delta after every checkpoint

["Martijn Visser (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/FLINK-39892?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Martijn Visser updated FLINK-39892:
-----------------------------------
    Fix Version/s: kafka-5.1.0
                       (was: 2.3.0)

> Exactly-once kafka sink reports a negative numBytesOut delta after every checkpoint
> -----------------------------------------------------------------------------------
>
>                 Key: FLINK-39892
>                 URL: https://issues.apache.org/jira/browse/FLINK-39892
>             Project: Flink
>          Issue Type: Bug
>          Components: Connectors / Kafka
>    Affects Versions: 2.2.1
>            Reporter: Efrat Levitan
>            Assignee: Efrat Levitan
>            Priority: Major
>              Labels: pull-request-available
>             Fix For: kafka-5.1.0
>
>
> The following counters
> {{taskmanager.job.task.numBytesOutPerSecond}}
> {{taskmanager.job.task.numBytesOut}}
> {{taskmanager.job.task.operator.numBytesOutPerSecond}}
> {{taskmanager.job.task.operator.numBytesSend}}
> frequently report a value decrement for exactly once kafka sink.
> This is because {{KafkaWriter#initKafkaMetrics}} is called per producer and resets {{byteOutMetric}} but not {{latestOutgoingByteTotal}}
> {{KafkaWriter#registerMetricSync}} calculates the delta of [outgoingBytesUntilNow - latestOutgoingByteTotal|https://github.com/apache/flink-connector-kafka/blob/main/flink-connector-kafka/src/main/java/org/apache/flink/connector/kafka/sink/KafkaWriter.java#L276-L278] so every time a new producer is spawned, {{numBytesOutCounter.inc}} is called with a negative delta, (0 - {{latestOutgoingByteTotal}} of the former producer)
> Different exporters handle this differently - otel exporter [drops|https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-otel/src/main/java/org/apache/flink/metrics/otel/OpenTelemetryMetricAdapter.java#L66-L73] the datapoint with a warn
> While prometheus exporter treats flink counters as [gauges|https://github.com/apache/flink/blob/master/flink-metrics/flink-metrics-prometheus/src/main/java/org/apache/flink/metrics/prometheus/AbstractPrometheusReporter.java#L210-L212] so the incorrect datapoint is still reported.



--
This message was sent by Atlassian Jira
(v10.3.26#10030026)