You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Shuai Lin (JIRA)" <ji...@apache.org> on 2017/04/06 00:23:41 UTC
[jira] [Commented] (KAFKA-5010) Log cleaner crashed with
BufferOverflowException when writing to the writeBuffer
[ https://issues.apache.org/jira/browse/KAFKA-5010?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15958091#comment-15958091 ]
Shuai Lin commented on KAFKA-5010:
----------------------------------
Hi [~ijuma], I'm not so familiar with kafka internals enough that I'm afraid I can't find a fix myself. Do you think someone in the community can work on this?
> Log cleaner crashed with BufferOverflowException when writing to the writeBuffer
> --------------------------------------------------------------------------------
>
> Key: KAFKA-5010
> URL: https://issues.apache.org/jira/browse/KAFKA-5010
> Project: Kafka
> Issue Type: Bug
> Components: log
> Affects Versions: 0.10.2.0
> Reporter: Shuai Lin
> Priority: Critical
> Labels: reliability
> Fix For: 0.11.0.0
>
>
> After upgrading from 0.10.0.1 to 0.10.2.0 the log cleaner thread crashed with BufferOverflowException when writing the filtered records into the writeBuffer:
> {code}
> [2017-03-24 10:41:03,926] INFO [kafka-log-cleaner-thread-0], Starting (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,177] INFO Cleaner 0: Beginning cleaning of log app-topic-20170317-20. (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,177] INFO Cleaner 0: Building offset map for app-topic-20170317-20... (kafka.log.LogCleaner)
> [2017-03-24 10:41:04,387] INFO Cleaner 0: Building offset map for log app-topic-20170317-20 for 1 segments in offset range [9737795, 9887707). (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,101] INFO Cleaner 0: Offset map for log app-topic-20170317-20 complete. (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,106] INFO Cleaner 0: Cleaning log app-topic-20170317-20 (cleaning prior to Fri Mar 24 10:36:06 GMT 2017, discarding tombstones prior to Thu Mar 23 10:18:02 GMT 2017)... (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,110] INFO Cleaner 0: Cleaning segment 0 in log app-topic-20170317-20 (largest timestamp Fri Mar 24 09:58:25 GMT 2017) into 0, retaining deletes. (kafka.log.LogCleaner)
> [2017-03-24 10:41:07,372] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner)
> java.nio.BufferOverflowException
> at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
> at org.apache.kafka.common.record.LogEntry.writeTo(LogEntry.java:98)
> at org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:158)
> at org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:111)
> at kafka.log.Cleaner.cleanInto(LogCleaner.scala:468)
> at kafka.log.Cleaner.$anonfun$cleanSegments$1(LogCleaner.scala:405)
> at kafka.log.Cleaner.$anonfun$cleanSegments$1$adapted(LogCleaner.scala:401)
> at scala.collection.immutable.List.foreach(List.scala:378)
> at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:401)
> at kafka.log.Cleaner.$anonfun$clean$6(LogCleaner.scala:363)
> at kafka.log.Cleaner.$anonfun$clean$6$adapted(LogCleaner.scala:362)
> at scala.collection.immutable.List.foreach(List.scala:378)
> at kafka.log.Cleaner.clean(LogCleaner.scala:362)
> at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:241)
> at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:220)
> at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
> [2017-03-24 10:41:07,375] INFO [kafka-log-cleaner-thread-0], Stopped (kafka.log.LogCleaner)
> {code}
> I tried different values of log.cleaner.buffer.size, from 512K to 2M to 10M to 128M, all with no luck: The log cleaner thread crashed immediately after the broker got restarted. But setting it to 256MB fixed the problem!
> Here are the settings for the cluster:
> {code}
> - log.message.format.version = 0.9.0.0 (we use 0.9 format because have old consumers)
> - log.cleaner.enable = 'true'
> - log.cleaner.min.cleanable.ratio = '0.1'
> - log.cleaner.threads = '1'
> - log.cleaner.io.buffer.load.factor = '0.98'
> - log.roll.hours = '24'
> - log.cleaner.dedupe.buffer.size = 2GB
> - log.segment.bytes = 256MB (global is 512MB, but we have been using 256MB for this topic)
> - message.max.bytes = 10MB
> {code}
> Given that the size of readBuffer and writeBuffer are exactly the same (half of log.cleaner.io.buffer.size), why would the cleaner throw a BufferOverflowException when writing the filtered records into the writeBuffer? IIUC that should never happen because the size of the filtered records should be no greater that the size of the readBuffer, thus no greater than the size of the writeBuffer.
--
This message was sent by Atlassian JIRA
(v6.3.15#6346)