[jira] [Created] (KAFKA-20711) Streams task restore-remaining-records metric invalid under EOS

["Nicholas Telford (Jira)" <ji...@apache.org>]

Nicholas Telford created KAFKA-20711:
----------------------------------------

             Summary: Streams task restore-remaining-records metric invalid under EOS
                 Key: KAFKA-20711
                 URL: https://issues.apache.org/jira/browse/KAFKA-20711
             Project: Kafka
          Issue Type: Bug
          Components: streams
    Affects Versions: 4.3.0
            Reporter: Nicholas Telford
            Assignee: Nicholas Telford


The Kafka Streams Task-level metric {{restore-remaining-records}} is intended to track the total number of records that still need to be restored.

When the application runs under EOS, this metric is inaccurate, never actually dropping to 0 for fully restored tasks, and always showing values substantially higher than reality.

The root-cause is that the metric is initialized with a total number of records to restored derived as {{logEndOffset - committedOffset}}, using a READ_UNCOMMITTED consumer.

This offset range naturally includes uncommitted records and transaction markers, in addition to the actual records to restore.

When decrementing the metric during restore, we decrement by the actual number of (committed) records that were restored. Since this excludes uncommitted records and transaction markers, we will never decrement the metric by the total it was initialized with.

I have a fix that I will raise a PR for.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)