[jira] [Created] (KAFKA-13447) Consumer should not reuse committed offset after topic recreation

["Jason Gustafson (Jira)" <ji...@apache.org>]

Jason Gustafson created KAFKA-13447:
---------------------------------------

             Summary: Consumer should not reuse committed offset after topic recreation
                 Key: KAFKA-13447
                 URL: https://issues.apache.org/jira/browse/KAFKA-13447
             Project: Kafka
          Issue Type: Bug
            Reporter: Jason Gustafson


KAFKA-12257 fixes an issue in which the consumer is unable to make progress after a topic has been recreated. The problem was that the client could not distinguish between stale metadata with a lower leader epoch and a recreated topic with a lower leader epoch. With TopicId support in KIP-516, the client is able to tell when a topic has been recreated since the new topic will have a different ID. 

However, what the patch did not fix is the potential reuse of the current offset position on the recreated topic. For example, say that the consumer is at offset N when the topic gets recreated. Currently, the consumer will continue fetching from offset N after detecting the recreation. The most likely result of this is either an offset out of range error or a log truncation error, but it is also possible for the offset position to remain valid on the recreated topic (say for a low-volume topic where the offsets is already low, or a case where the consumer was down for a while). 

To fix this issue completely, we need to store the topicId along with the committed offset in __consumer_offsets. This would allow the consumer to detect when the offset is no longer relevant for the current topic. We also need to decide how to raise this case to the user. If the user has enabled automatic offset reset, we can probably use that. Otherwise, we might need a new exception type to signal the user that the position needs to be reset.





--
This message was sent by Atlassian Jira
(v8.20.1#820001)