[jira] [Updated] (KAFKA-20711) Streams task restore-remaining-records metric invalid under EOS

["Nicholas Telford (Jira)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-20711?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Nicholas Telford updated KAFKA-20711:
-------------------------------------
    Priority: Minor  (was: Major)

> Streams task restore-remaining-records metric invalid under EOS
> ---------------------------------------------------------------
>
>                 Key: KAFKA-20711
>                 URL: https://issues.apache.org/jira/browse/KAFKA-20711
>             Project: Kafka
>          Issue Type: Bug
>          Components: streams
>    Affects Versions: 4.3.0
>            Reporter: Nicholas Telford
>            Assignee: Nicholas Telford
>            Priority: Minor
>
> The Kafka Streams Task-level metric {{restore-remaining-records}} is intended to track the total number of records that still need to be restored.
> When the application runs under EOS, this metric is inaccurate, never actually dropping to 0 for fully restored tasks, and always showing values substantially higher than reality.
> The root-cause is that the metric is initialized with a total number of records to restored derived as {{logEndOffset - committedOffset}}, using a READ_UNCOMMITTED consumer.
> This offset range naturally includes uncommitted records and transaction markers, in addition to the actual records to restore.
> When decrementing the metric during restore, we decrement by the actual number of (committed) records that were restored. Since this excludes uncommitted records and transaction markers, we will never decrement the metric by the total it was initialized with.
> I have a fix that I will raise a PR for.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)