You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Victor Garcia (Jira)" <ji...@apache.org> on 2021/06/09 23:56:03 UTC
[jira] [Commented] (KAFKA-10501) Log Cleaner never clean up some
__consumer_offsets partitions
[ https://issues.apache.org/jira/browse/KAFKA-10501?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17360469#comment-17360469 ]
Victor Garcia commented on KAFKA-10501:
---------------------------------------
[~mbaluta] do you think this issue is the same as explained in this comment? https://issues.apache.org/jira/browse/KAFKA-8335?focusedCommentId=16870892&page=com.atlassian.jira.plugin.system.issuetabpanels%3Acomment-tabpanel#comment-16870892
I wonder how you got the content of the example, is this just the dump of a log file or you were just consuming from the topic?
When i dump the log file i don't see those ABORT and COMMIT messages
> Log Cleaner never clean up some __consumer_offsets partitions
> -------------------------------------------------------------
>
> Key: KAFKA-10501
> URL: https://issues.apache.org/jira/browse/KAFKA-10501
> Project: Kafka
> Issue Type: Bug
> Components: log, log cleaner
> Affects Versions: 2.5.0
> Reporter: Mykhailo Baluta
> Priority: Major
>
> Some __consumer_offsets partitions contain "broken" messages in the second log segment.
> Example:
> {code:java}
> offset: 745253728 position: 49793647 CreateTime: 1594539245536 isvalid: true keysize: 99 valuesize: 28 magic: 2 compresscodec: NONE producerId: 37146 producerEpoch: 0 sequence: 0 isTransactional: true headerKeys: []
> offset: 745253729 position: 49793844 CreateTime: 1594539245548 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 37146 producerEpoch: 0 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: COMMIT coordinatorEpoch: 59
> offset: 745256523 position: 50070884 CreateTime: 1594540927673 isvalid: true keysize: 4 valuesize: 6 magic: 2 compresscodec: NONE producerId: 37146 producerEpoch: 1 sequence: -1 isTransactional: true headerKeys: [] endTxnMarker: ABORT coordinatorEpoch: 59
> offset: 745256543 position: 50073185 CreateTime: 1594541667798 isvalid: true keysize: 99 valuesize: 28 magic: 2 compresscodec: NONE producerId: 37146 producerEpoch: 0 sequence: 0 isTransactional: true headerKeys: []
> {code}
> Seems like the last 2 records are stored in the wrong order. As a result the last message is transactional and not any ABORT/COMMIT message after. It leads to a producer state with ongoing transactions and firstUncleanableDirtyOffset = 745256543. Thus, compaction always skips for such topic partitions.
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)