You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Lucas Bradstreet (Jira)" <ji...@apache.org> on 2021/01/11 19:14:00 UTC

[jira] [Created] (KAFKA-12177) Retention is not idempotent

Lucas Bradstreet created KAFKA-12177:
----------------------------------------

             Summary: Retention is not idempotent
                 Key: KAFKA-12177
                 URL: https://issues.apache.org/jira/browse/KAFKA-12177
             Project: Kafka
          Issue Type: Improvement
            Reporter: Lucas Bradstreet


Kafka today applies retention in the following order:
 # Time
 # Size
 # Log start offset


Today it is possible for a segment with offsets less than the log start offset to contain data that is not deletable due to time retention. This means that it's possible for log start offset retention to unblock further deletions as a result of time based retention. Note that this does require a case where the max timestamp for each segment increases, decreases and then increases again. Even so it would be nice to make retention idempotent by applying log start offset retention first, followed by size and time. This would also be potentially cheaper to perform as neither log start offset and size retention require the maxTimestamp for a segment to be loaded from disk after a broker restart.



--
This message was sent by Atlassian Jira
(v8.3.4#803005)