You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "William Yu (JIRA)" <ji...@apache.org> on 2017/01/24 19:14:26 UTC
[jira] [Commented] (KAFKA-3894) LogCleaner thread crashes if not
even one segment can fit in the offset map
[ https://issues.apache.org/jira/browse/KAFKA-3894?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15836473#comment-15836473 ]
William Yu commented on KAFKA-3894:
-----------------------------------
I know this bug is resolved, but we just encountered this bug in our 0.9.0.1 Cluster.
{code}
[2017-01-24 17:17:30,035] ERROR [kafka-log-cleaner-thread-0], Error due to (kafka.log.LogCleaner)
java.lang.IllegalArgumentException: requirement failed: 13042136566 messages in segment __consumer_offsets-32/00000000000000000000.log but offset map can fit only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease log.cleaner.threads
{code}
We've tried the following attempts to fix:
- Increase the log.cleaner.dedupe.buffer.size but the # of messages is greater then MAX_INT and will be beyond the amount of memory we can allocated.
- Wipe away the partition in question from a single broker and let the data replicate back. Did not work as all the replicas for this partition also have an issue where they cannot compact the topic.
Does anyone else know of another solution to recover this partition? Do I need to just wipe away the whole partition completely?
> LogCleaner thread crashes if not even one segment can fit in the offset map
> ---------------------------------------------------------------------------
>
> Key: KAFKA-3894
> URL: https://issues.apache.org/jira/browse/KAFKA-3894
> Project: Kafka
> Issue Type: Bug
> Components: core
> Affects Versions: 0.8.2.2, 0.9.0.1, 0.10.0.0
> Environment: Oracle JDK 8
> Ubuntu Precise
> Reporter: Tim Carey-Smith
> Assignee: Tom Crayford
> Labels: compaction
> Fix For: 0.10.1.0
>
>
> The log-cleaner thread can crash if the number of keys in a topic grows to be too large to fit into the dedupe buffer.
> The result of this is a log line:
> {quote}
> broker=0 pri=ERROR t=kafka-log-cleaner-thread-0 at=LogCleaner \[kafka-log-cleaner-thread-0\], Error due to java.lang.IllegalArgumentException: requirement failed: 9750860 messages in segment MY_FAVORITE_TOPIC-2/00000000000047580165.log but offset map can fit only 5033164. You can increase log.cleaner.dedupe.buffer.size or decrease log.cleaner.threads
> {quote}
> As a result, the broker is left in a potentially dangerous situation where cleaning of compacted topics is not running.
> It is unclear if the broader strategy for the {{LogCleaner}} is the reason for this upper bound, or if this is a value which must be tuned for each specific use-case.
> Of more immediate concern is the fact that the thread crash is not visible via JMX or exposed as some form of service degradation.
> Some short-term remediations we have made are:
> * increasing the size of the dedupe buffer
> * monitoring the log-cleaner threads inside the JVM
--
This message was sent by Atlassian JIRA
(v6.3.4#6332)