[jira] [Created] (KAFKA-12889) log clean group consider empty log segment to avoid empty log left

["qiang Liu (Jira)" <ji...@apache.org>]

qiang Liu created KAFKA-12889:
---------------------------------

             Summary: log clean group consider empty log segment to avoid empty log left
                 Key: KAFKA-12889
                 URL: https://issues.apache.org/jira/browse/KAFKA-12889
             Project: Kafka
          Issue Type: Improvement
          Components: log cleaner
    Affects Versions: 0.10.1.1
            Reporter: qiang Liu


to avoid log index 4 byte relative offset overflow, log cleaner group check log segments offset to make sure group offset range not exceed Int.MaxValue.

this offset check currentlly not cosider next is next log segment is empty, so there will left empty log files every about 2^31 messages.

the left empty logs will be reprocessed every clean cycle, which will rewrite it with same empty content, witch cause little no need io.

for __consumer_offsets topic, normally we can set cleanup.policy to compact,delete to get rid of this.

my cluster is 0.10.1.1, but after aylize trunk code, it should has same problem too.

 

some of my left empty logs,(run ls -l)

-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.index
-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.log
-rw-r----- 1 u g 0 Dec 16 2017 00000000000000000000.timeindex
-rw-r----- 1 u g 0 Jan  15 2018 00000000002148249632.index
-rw-r----- 1 u g 0 Jan  15 2018 00000000002148249632.log
-rw-r----- 1 u g 0 Jan  15 2018 00000000002148249632.timeindex
-rw-r----- 1 u g 0 Jan  27 2018 00000000004295766494.index
-rw-r----- 1 u g 0 Jan  27 2018 00000000004295766494.log
-rw-r----- 1 u g 0 Jan  27 2018 00000000004295766494.timeindex

 



--
This message was sent by Atlassian Jira
(v8.3.4#803005)