[jira] [Commented] (KAFKA-3330) Truncate log cleaner offset checkpoint if the log is truncated

["James Cheng (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-3330?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15269671#comment-15269671 ] 

James Cheng commented on KAFKA-3330:
------------------------------------

We ran into this today. While we are waiting for the 0.10 release, is there anything we can do to work around this bug?

> Truncate log cleaner offset checkpoint if the log is truncated
> --------------------------------------------------------------
>
>                 Key: KAFKA-3330
>                 URL: https://issues.apache.org/jira/browse/KAFKA-3330
>             Project: Kafka
>          Issue Type: Bug
>            Reporter: Dong Lin
>            Assignee: Dong Lin
>            Priority: Critical
>             Fix For: 0.10.0.0
>
>
> Were getting a number of failures of the log compaction thread with the
> following error:
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Beginning cleaning of log
> __consumer_offsets-93.
> 2016/02/02 00:13:58.832 [LogCleaner] Cleaner 0: Building offset map for
> __consumer_offsets-93...
> 2016/02/02 00:13:59.048 [LogCleaner] Cleaner 0: Building offset map for log
> __consumer_offsets-93 for 2 segments in offset range [11951210572,
> 11952632314).
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Error
> due to
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 11951210572 but segment base offset is 11950300163 for log
> __consumer_offsets-93.
> at scala.Predef$.require(Predef.scala:233) ~[scala-library-2.10.4.jar:?]
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:561)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.Cleaner.clean(LogCleaner.scala:306)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:217)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:195)
> ~[kafka_2.10-0.8.2.56.jar:?]
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> ~[kafka_2.10-0.8.2.56.jar:?]
> 2016/02/02 00:13:59.066 [LogCleaner] [kafka-log-cleaner-thread-0], Stopped
> We found that this may be caused in the following scenario:
> - we have three log segments with offset range [100, 200), [200, 300), and [300, 400) respectively. 300 is the base offset of the active log segment. Log cleaner offset checkpoint is also 300.
> - After log is truncated to offset 220, the log segments become [100, 200), [200, 220). The Log cleaner offset checkpoint is still 300.
> - After new messages are appended to the log, the log segments become [100, 200), [200, 320), [320, 420). The Log cleaner offset checkpoint is still 300.
> - Log cleaner cleans the log starting at offset 300. The require(offset == start) in Cleaner.buildOffsetMap() fails because the the offset 300 is not the base offset of any segments.
> To fix the problem, when the log is truncated to an offset smaller than cleaner offset checkpoint, we should reset cleaner offset checkpoint to the base offset of the active segment if this value is larger than the checkpointed offset.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)