[jira] [Created] (KAFKA-5010) Log cleaner crashed with BufferOverflowException when writing to the writeBuffer

["Shuai Lin (JIRA)" <ji...@apache.org>]

Shuai Lin created KAFKA-5010:
--------------------------------

             Summary: Log cleaner crashed with BufferOverflowException when writing to the writeBuffer
                 Key: KAFKA-5010
                 URL: https://issues.apache.org/jira/browse/KAFKA-5010
             Project: Kafka
          Issue Type: Bug
          Components: log
    Affects Versions: 0.10.2.0
            Reporter: Shuai Lin


After upgrading from 0.10.0.1 to 0.10.2.0 the log cleaner thread crashed with BufferOverflowException when writing the filtered records into the writeBuffer:

{code}
[2017-03-24 10:41:03,926] INFO [kafka-log-cleaner-thread-0], Starting  (kafka.log.LogCleaner)
[2017-03-24 10:41:04,177] INFO Cleaner 0: Beginning cleaning of log app-topic-20170317-20. (kafka.log.LogCleaner)
[2017-03-24 10:41:04,177] INFO Cleaner 0: Building offset map for app-topic-20170317-20... (kafka.log.LogCleaner)
[2017-03-24 10:41:04,387] INFO Cleaner 0: Building offset map for log app-topic-20170317-20 for 1 segments in offset range [9737795, 9887707). (kafka.log.LogCleaner)
[2017-03-24 10:41:07,101] INFO Cleaner 0: Offset map for log app-topic-20170317-20 complete. (kafka.log.LogCleaner)
[2017-03-24 10:41:07,106] INFO Cleaner 0: Cleaning log app-topic-20170317-20 (cleaning prior to Fri Mar 24 10:36:06 GMT 2017, discarding tombstones prior to Thu Mar 23 10:18:02 GMT 2017)... (kafka.log.LogCleaner)
[2017-03-24 10:41:07,110] INFO Cleaner 0: Cleaning segment 0 in log app-topic-20170317-20 (largest timestamp Fri Mar 24 09:58:25 GMT 2017) into 0, retaining deletes. (kafka.log.LogCleaner)
[2017-03-24 10:41:07,372] ERROR [kafka-log-cleaner-thread-0], Error due to  (kafka.log.LogCleaner)
java.nio.BufferOverflowException
        at java.nio.HeapByteBuffer.put(HeapByteBuffer.java:206)
        at org.apache.kafka.common.record.LogEntry.writeTo(LogEntry.java:98)
        at org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:158)
        at org.apache.kafka.common.record.MemoryRecords.filterTo(MemoryRecords.java:111)
        at kafka.log.Cleaner.cleanInto(LogCleaner.scala:468)
        at kafka.log.Cleaner.$anonfun$cleanSegments$1(LogCleaner.scala:405)
        at kafka.log.Cleaner.$anonfun$cleanSegments$1$adapted(LogCleaner.scala:401)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at kafka.log.Cleaner.cleanSegments(LogCleaner.scala:401)
        at kafka.log.Cleaner.$anonfun$clean$6(LogCleaner.scala:363)
        at kafka.log.Cleaner.$anonfun$clean$6$adapted(LogCleaner.scala:362)
        at scala.collection.immutable.List.foreach(List.scala:378)
        at kafka.log.Cleaner.clean(LogCleaner.scala:362)
        at kafka.log.LogCleaner$CleanerThread.cleanOrSleep(LogCleaner.scala:241)
        at kafka.log.LogCleaner$CleanerThread.doWork(LogCleaner.scala:220)
        at kafka.utils.ShutdownableThread.run(ShutdownableThread.scala:63)
[2017-03-24 10:41:07,375] INFO [kafka-log-cleaner-thread-0], Stopped  (kafka.log.LogCleaner)
{code}

I tried different values of log.cleaner.buffer.size, from 512K to 2M to 10M to 128M, all with no luck: The log cleaner thread crashed immediately after the broker got restarted. But setting it to 256MB fixed the problem!

Here are the settings for the cluster:
{code}
- log.message.format.version = 0.9.0.0 (we use 0.9 format because have old consumers)
- log.cleaner.enable = 'true'
- log.cleaner.min.cleanable.ratio = '0.1'
- log.cleaner.threads = '1'
- log.cleaner.io.buffer.load.factor = '0.98'
- log.roll.hours = '24'
- log.cleaner.dedupe.buffer.size = 2GB 
- log.segment.bytes = 256MB (global is 512MB, but we have been using 256MB for this topic)
- message.max.bytes = 10MB
{code}

Given that the size of readBuffer and writeBuffer are exactly the same (half of log.cleaner.io.buffer.size), why would the cleaner throw a BufferOverflowException when writing the filtered records into the writeBuffer? IIUC that should never happen because the size of the filtered records should be no greater that the size of the readBuffer, thus no greater than the size of the writeBuffer.





--
This message was sent by Atlassian JIRA
(v6.3.15#6346)