[jira] [Created] (KAFKA-18216) High water mark or last stable offset aren't always updated after a fetch request is completed

["Philip Nee (Jira)" <ji...@apache.org>]

Philip Nee created KAFKA-18216:
----------------------------------

             Summary: High water mark or last stable offset aren't always updated after a fetch request is completed
                 Key: KAFKA-18216
                 URL: https://issues.apache.org/jira/browse/KAFKA-18216
             Project: Kafka
          Issue Type: Improvement
          Components: clients, consumer
            Reporter: Philip Nee


We've noticed AsyncKafkaConsumer doesn't always update the high water mark/LSO followed by handling a successful fetch response. And we know consumer lag metrics is calculated by HWM/LSO - current fetched position.  We are suspecting this could have a subtle effect into how consumer lag is recorded, which might have a slight impact into the accuracy of client metrics reporting.

The consumer records consumer lag when reading the fetched record 

The consumer updates the HWM/LSO when the background thread completes the fetched request.

In the original implementation, the fetcher consistently updates the HWM/LSO after handling the completed fetch request.

In the new implementation, due to the async threading model, we can't guarantee the sequence of the event.

This defect is affecting neither performance nor correctness and is therefore marked as "Minor"

 

This can be easily reproduced using the java-produce-consumer-demo.sh example.  Ensure to produce enough records (I use 200000000 records, less is fine as well).  Custom logging is required.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)