You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Prashant (JIRA)" <ji...@apache.org> on 2019/08/13 07:10:00 UTC
[jira] [Updated] (KAFKA-8795) Kafka - Segment deleted earlier than
expected
[ https://issues.apache.org/jira/browse/KAFKA-8795?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Prashant updated KAFKA-8795:
----------------------------
Component/s: core
> Kafka - Segment deleted earlier than expected
> ---------------------------------------------
>
> Key: KAFKA-8795
> URL: https://issues.apache.org/jira/browse/KAFKA-8795
> Project: Kafka
> Issue Type: Bug
> Components: core, log cleaner
> Affects Versions: 1.0.0
> Environment: production
> Reporter: Prashant
> Priority: Critical
>
> Hi
> We are observing that log segments are getting deleted prematurely even before largest message timestamp in segment has not yet reached its retention period.
> We are using broker version kafka_2.11-1.0.0.
> Looking at timeindex file , I see that it is getting append with "timestamp: 0" entries with offset= start offset of segment.
> Example :
> File : 00000000000000047730.timeindex
> File : 00000000000000047730.timeindex
> timestamp: 1565565117007 offset: 47799
> timestamp: 1565565117037 offset: 47846
> timestamp: 1565565117087 offset: 47917
> ...
> timestamp: 1565565118742 offset: 50607
> *timestamp: 0* offset: 47730
> timestamp: 0 offset: 47730
>
> Last message published to this segment was at 1565565118742 which is 00:11 BST and segment got deleted by log cleaner at 12:12 BST.
>
>
> [12:12:34,143] INFO Rolled new log segment for 'TOPICNAME-0' in 50 ms. (kafka.log.Log)
> [12:12:34,143] INFO Scheduling log segment 47942 for log TOPICNAME-0 for deletion. (kafka.log.Log)
> [12:12:34,147] INFO Incrementing log start offset of partition TOPICNAME-0 to 50749 in dir /home/test/data-5 (kafka.log.Log)
> [12:12:34,149] INFO Cleared earliest 0 entries from epoch cache based on passed offset 50749 leaving 1 in EpochFile for partition TOPICNAME-0 (kafka.server.epoch.LeaderEpochFileCache)
> [12:13:34,147] INFO Deleting segment 47942 from log TOPICNAME-0. (kafka.log.Log)
> [12:13:34,148] INFO Deleting index /home/test/data-5/TOPICNAME-0/00000000000000047942.index.deleted (kafka.log.OffsetIndex)
> [12:13:34,149] INFO Deleting index /home/test/data-5/TOPICNAME-0/00000000000000047942.timeindex.deleted (kafka.log.TimeIndex)
>
> Please help us understand what is causing this issue and how to fix it.
>
--
This message was sent by Atlassian JIRA
(v7.6.14#76016)