[jira] [Created] (KAFKA-6854) Log cleaner fails with transaction markers that are deleted during clean

["Rajini Sivaram (JIRA)" <ji...@apache.org>]

Rajini Sivaram created KAFKA-6854:
-------------------------------------

             Summary: Log cleaner fails with transaction markers that are deleted during clean
                 Key: KAFKA-6854
                 URL: https://issues.apache.org/jira/browse/KAFKA-6854
             Project: Kafka
          Issue Type: Task
          Components: core
    Affects Versions: 1.1.0
            Reporter: Rajini Sivaram
            Assignee: Rajini Sivaram


Log cleaner grows buffers when `result.messagesRead` is zero. In a typical scenario, this is a result of source buffer being too small to read the first batch. The buffer is then doubled in size until one batch can be read, up to a maximum of `max.message.size`. There are issues with the maximum message size used in calculations as reported in KAFKA-6834. But there is a separate issue with the use of `result.messagesRead` when transactions are used. This contains the number of filtered messages read from source which can be zero when a transaction control marker is discarded. Log cleaner incorrectly assumes that messages were not read because the buffer was too small. This can result in the log cleaner attempting to grow buffers forever, failing with an exception when `max.message.bytes` is reached. This kills the log cleaner.

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)