[jira] [Commented] (KAFKA-6264) Log cleaner thread may die on legacy segment containing messages whose offsets are too large

["ASF GitHub Bot (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-6264?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=16466651#comment-16466651 ] 

ASF GitHub Bot commented on KAFKA-6264:
---------------------------------------

dhruvilshah3 opened a new pull request #4975: KAFKA-6264: Log cleaner thread may die on legacy segment containing messages whose offsets are too large
URL: https://github.com/apache/kafka/pull/4975
 
 
   Implementation for splitting log segments with offset overflow.
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


> Log cleaner thread may die on legacy segment containing messages whose offsets are too large
> --------------------------------------------------------------------------------------------
>
>                 Key: KAFKA-6264
>                 URL: https://issues.apache.org/jira/browse/KAFKA-6264
>             Project: Kafka
>          Issue Type: Bug
>          Components: core
>    Affects Versions: 0.10.2.1, 1.0.0, 0.11.0.2
>            Reporter: Jiangjie Qin
>            Assignee: Dhruvil Shah
>            Priority: Critical
>             Fix For: 2.0.0
>
>
> We encountered a problem that some of the legacy log segments contains messages whose offsets are larger than {{SegmentBaseOffset + Int.MaxValue}}.
> Prior to 0.10.2.0, we do not assert the offset of the messages when appending them to the log segments. Due to KAFKA-5413, the log cleaner may append messages whose offset is greater than {{base_offset + Int.MaxValue}} into the segment during the log compaction.
> After the brokers are upgraded, those log segments cannot be compacted anymore because the compaction will fail immediately due to the offset range assertion we added to the LogSegment.
> We have seen this issue in the __consumer_offsets topic so it could be a general problem. There is no easy solution for the users to recover from this case. 
> One solution is to split such log segments in the log cleaner once it sees a message with problematic offset and append those messages to a separate log segment with a larger base_offset.
> Due to the impact of the issue. We may want to consider backporting the fix to previous affected versions.



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)