[jira] [Created] (KAFKA-6347) Starting offset breach based log segment deletion never considers active segment

["Guozhang Wang (JIRA)" <ji...@apache.org>]

Guozhang Wang created KAFKA-6347:
------------------------------------

             Summary: Starting offset breach based log segment deletion never considers active segment
                 Key: KAFKA-6347
                 URL: https://issues.apache.org/jira/browse/KAFKA-6347
             Project: Kafka
          Issue Type: Improvement
          Components: core
            Reporter: Guozhang Wang


This observation is related to KIP-107: in {{Log#deleteLogStartOffsetBreachedSegments}}, we designed the predicate to enforce that the current segment can only be considered for deletion, when it 1) has the next segment, 2) the next segment's starting offset is no larger than the log start offset.

{code}
def shouldDelete(segment: LogSegment, nextSegmentOpt: Option[LogSegment]) =
      nextSegmentOpt.exists(_.baseOffset <= logStartOffset)
{code}

This means that, the current active segment would never be considered for purging. Normally this is OK given that we will eventually roll out a new segment. However, with the default size of segment of 1GB it means that with the purge data API we are in the worst case not being able to purge the data for up to 1GB per topic partition.



--
This message was sent by Atlassian JIRA
(v6.4.14#64029)