[jira] [Commented] (KAFKA-4529) tombstone may be removed earlier than it should

["Jun Rao (JIRA)" <ji...@apache.org>]

    [ https://issues.apache.org/jira/browse/KAFKA-4529?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=15743987#comment-15743987 ] 

Jun Rao commented on KAFKA-4529:
--------------------------------

[~becket_qin], do you think you have time to fix this issue? The easiest thing is to calculate deleteHorizonMs in the old way based on the last modified time of the segment?

> tombstone may be removed earlier than it should
> -----------------------------------------------
>
>                 Key: KAFKA-4529
>                 URL: https://issues.apache.org/jira/browse/KAFKA-4529
>             Project: Kafka
>          Issue Type: Bug
>    Affects Versions: 0.10.1.0
>            Reporter: Jun Rao
>             Fix For: 0.10.1.1
>
>
> As part of KIP-33, we introduced a regression on how tombstone is removed in a compacted topic. We want to delay the removal of a tombstone to avoid the case that a reader first reads a non-tombstone message on a key and then doesn't see the tombstone for the key because it's deleted too quickly. So, a tombstone is supposed to only be removed from a compacted topic after the tombstone is part of the cleaned portion of the log after delete.retention.ms.
> Before KIP-33, deleteHorizonMs in LogCleaner is calculated based on the last modified time, which is monotonically increasing from old to new segments. With KIP-33, deleteHorizonMs is calculated based on the message timestamp, which is not necessarily monotonically increasing.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)