You are viewing a plain text version of this content. The canonical link for it is here.
Posted to dev@kafka.apache.org by "Jan Filipiak (JIRA)" <ji...@apache.org> on 2017/10/12 04:46:00 UTC
[jira] [Created] (KAFKA-6056) LogCleaner always cleaning into 1
Segment might exeed relativ offset range
Jan Filipiak created KAFKA-6056:
-----------------------------------
Summary: LogCleaner always cleaning into 1 Segment might exeed relativ offset range
Key: KAFKA-6056
URL: https://issues.apache.org/jira/browse/KAFKA-6056
Project: Kafka
Issue Type: Bug
Components: core, log
Affects Versions: 0.11.0.0
Reporter: Jan Filipiak
Priority: Minor
After having an Issue with compaction stopping for some time. It can be an issue that the LogCleaner will always clean into 1 Segment per sizegroup.
Usually the Log enforces a maximum distance between min and max offset in a LogSegment. If that Distance would be exeeded in maybeRoll() a new logsegment would be rolled. I assume this is because relative offset might be stored as integer. The LogCleaner OTOH is not going to roll a new LogSegment as its only ever using 1 Segment to clean into.
A lenghty discussion about this can be found in the slack community:
https://confluentcommunity.slack.com/archives/C49R61XMM/p1506914441000005
The observed stacktrace is as follows:
https://gist.github.com/brettrann/ce52343692696a45d5b9f4df723bcd14
I could imagin also enfocing Integer.MAX_VALUE as offset distance in
groupSegmentsBySize in the LogCleaner to make sure a Segment doesnt exeed this limit.
--
This message was sent by Atlassian JIRA
(v6.4.14#64029)