You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "qiang Liu (Jira)" <ji...@apache.org> on 2021/06/04 07:13:00 UTC
[jira] [Commented] (KAFKA-12889) log clean group consider empty log
segment to avoid empty log left
[ https://issues.apache.org/jira/browse/KAFKA-12889?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=17357114#comment-17357114 ]
qiang Liu commented on KAFKA-12889:
-----------------------------------
create a pull request on github
> log clean group consider empty log segment to avoid empty log left
> ------------------------------------------------------------------
>
> Key: KAFKA-12889
> URL: https://issues.apache.org/jira/browse/KAFKA-12889
> Project: Kafka
> Issue Type: Improvement
> Components: log cleaner
> Affects Versions: 0.10.1.1
> Reporter: qiang Liu
> Priority: Minor
>
> to avoid log index 4 byte relative offset overflow, log cleaner group check log segments offset to make sure group offset range not exceed Int.MaxValue.
> this offset check currentlly not cosider next is next log segment is empty, so there will left empty log files every about 2^31 messages.
> the left empty logs will be reprocessed every clean cycle, which will rewrite it with same empty content, witch cause little no need io.
> for __consumer_offsets topic, normally we can set cleanup.policy to compact,delete to get rid of this.
> my cluster is 0.10.1.1, but after aylize trunk code, it should has same problem too.
>
> some of my left empty logs,(run ls -l)
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.index
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.log
> -rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.timeindex
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.index
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.log
> -rw-r----- 1 u g 0 Jan 15 2018 00000000002148249632.timeindex
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.index
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.log
> -rw-r----- 1 u g 0 Jan 27 2018 00000000004295766494.timeindex
>
--
This message was sent by Atlassian Jira
(v8.3.4#803005)