You are viewing a plain text version of this content. The canonical link for it is here.
Posted to users@kafka.apache.org by Rakesh Vidyadharan <rv...@gracenote.com> on 2016/04/27 22:14:20 UTC
How to work around log compaction error (0.8.2.2)
Hello,
We enabled log compaction on a few topics, as we want to preserve permanently the latest versions of messages published to specific topics. After enabling compaction, the log cleaner thread dies with the same error for the topics we tried it on. It looks like kafka has starting offset that does not exist in the topic (at least that is how I am reading the error). Any ideas on how we can work around this error?
Thanks
Rakesh
[2016-04-27 15:52:11,306] INFO [kafka-log-cleaner-thread-0], Starting (kafka.log.LogCleaner)
[2016-04-27 15:52:11,322] INFO Cleaner 0: Beginning cleaning of log metamorphosis.lineup-0. (kafka.log.LogCleaner)
[2016-04-27 15:52:11,323] INFO Cleaner 0: Building offset map for metamorphosis.lineup-0... (kafka.log.LogCleaner)
[2016-04-27 15:52:11,415] INFO Cleaner 0: Building offset map for log metamorphosis.lineup-0 for 1 segments in offset range [1553258, 2138466). (kafka.log.LogCleaner)
[2016-04-27 15:52:11,435] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: Last clean offset is 1553258 but segment base offset is 2125968 for log metamorphosis.lineup-0.
at scala.Predef$.require(Predef.scala:233)
at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509)
at kafka.log.Cleaner.clean(LogCleaner.scala:307)
at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221)
at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199)
at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
[2016-04-27 15:52:11,436] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner)
Re: How to work around log compaction error (0.8.2.2)
Posted by Rakesh Vidyadharan <rv...@gracenote.com>.
Hi Manikumar,
No, we are not using compression on our topics. I will try out Todd Palino’s suggestion regarding the offsets topic.
Thanks
Rakesh
On 27/04/2016 23:34, "Manikumar Reddy" <ma...@gmail.com> wrote:
>Hi,
>
> Are you enabling log compaction on a topic with compressed messages?
> If yes, then that might be the reason for the exception. 0.8.2.2 Log
>Compaction does
> not support compressed messages. This got fixed in 0.9.0.0 (KAFKA-1641,
>KAFKA-1374)
>
>Check below mail thread for some corrective actions
>http://grokbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops
>
>
>On Thu, Apr 28, 2016 at 1:44 AM, Rakesh Vidyadharan <
>rvidyadharan@gracenote.com> wrote:
>
>> Hello,
>>
>> We enabled log compaction on a few topics, as we want to preserve
>> permanently the latest versions of messages published to specific topics.
>> After enabling compaction, the log cleaner thread dies with the same error
>> for the topics we tried it on. It looks like kafka has starting offset
>> that does not exist in the topic (at least that is how I am reading the
>> error). Any ideas on how we can work around this error?
>>
>> Thanks
>> Rakesh
>>
>> [2016-04-27 15:52:11,306] INFO [kafka-log-cleaner-thread-0], Starting
>> (kafka.log.LogCleaner)
>> [2016-04-27 15:52:11,322] INFO Cleaner 0: Beginning cleaning of log
>> metamorphosis.lineup-0. (kafka.log.LogCleaner)
>> [2016-04-27 15:52:11,323] INFO Cleaner 0: Building offset map for
>> metamorphosis.lineup-0... (kafka.log.LogCleaner)
>> [2016-04-27 15:52:11,415] INFO Cleaner 0: Building offset map for log
>> metamorphosis.lineup-0 for 1 segments in offset range [1553258, 2138466).
>> (kafka.log.LogCleaner)
>> [2016-04-27 15:52:11,435] ERROR [kafka-log-cleaner-thread-0], Error due
>> to (kafka.log.LogCleaner)
>> java.lang.IllegalArgumentException: requirement failed: Last clean offset
>> is 1553258 but segment base offset is 2125968 for log
>> metamorphosis.lineup-0.
>> at scala.Predef$.require(Predef.scala:233)
>> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509)
>> at kafka.log.Cleaner.clean(LogCleaner.scala:307)
>> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221)
>> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199)
>> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
>> [2016-04-27 15:52:11,436] INFO [kafka-log-cleaner-thread-0], Stopped
>> (kafka.log.LogCleaner)
>>
>>
Re: How to work around log compaction error (0.8.2.2)
Posted by Manikumar Reddy <ma...@gmail.com>.
Hi,
Are you enabling log compaction on a topic with compressed messages?
If yes, then that might be the reason for the exception. 0.8.2.2 Log
Compaction does
not support compressed messages. This got fixed in 0.9.0.0 (KAFKA-1641,
KAFKA-1374)
Check below mail thread for some corrective actions
http://grokbase.com/t/kafka/users/159jbe18en/log-cleaner-thread-stops
On Thu, Apr 28, 2016 at 1:44 AM, Rakesh Vidyadharan <
rvidyadharan@gracenote.com> wrote:
> Hello,
>
> We enabled log compaction on a few topics, as we want to preserve
> permanently the latest versions of messages published to specific topics.
> After enabling compaction, the log cleaner thread dies with the same error
> for the topics we tried it on. It looks like kafka has starting offset
> that does not exist in the topic (at least that is how I am reading the
> error). Any ideas on how we can work around this error?
>
> Thanks
> Rakesh
>
> [2016-04-27 15:52:11,306] INFO [kafka-log-cleaner-thread-0], Starting
> (kafka.log.LogCleaner)
> [2016-04-27 15:52:11,322] INFO Cleaner 0: Beginning cleaning of log
> metamorphosis.lineup-0. (kafka.log.LogCleaner)
> [2016-04-27 15:52:11,323] INFO Cleaner 0: Building offset map for
> metamorphosis.lineup-0... (kafka.log.LogCleaner)
> [2016-04-27 15:52:11,415] INFO Cleaner 0: Building offset map for log
> metamorphosis.lineup-0 for 1 segments in offset range [1553258, 2138466).
> (kafka.log.LogCleaner)
> [2016-04-27 15:52:11,435] ERROR [kafka-log-cleaner-thread-0], Error due
> to (kafka.log.LogCleaner)
> java.lang.IllegalArgumentException: requirement failed: Last clean offset
> is 1553258 but segment base offset is 2125968 for log
> metamorphosis.lineup-0.
> at scala.Predef$.require(Predef.scala:233)
> at kafka.log.Cleaner.buildOffsetMap(LogCleaner.scala:509)
> at kafka.log.Cleaner.clean(LogCleaner.scala:307)
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:221)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:199)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:60)
> [2016-04-27 15:52:11,436] INFO [kafka-log-cleaner-thread-0], Stopped
> (kafka.log.LogCleaner)
>
>