[jira] [Resolved] (KAFKA-2871) Newly replicated brokers don't expire log segments properly

["Ismael Juma (JIRA)" <ji...@apache.org>]

     [ https://issues.apache.org/jira/browse/KAFKA-2871?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]

Ismael Juma resolved KAFKA-2871.
--------------------------------
    Resolution: Duplicate

This was changed as part of KIP-33.

> Newly replicated brokers don't expire log segments properly
> -----------------------------------------------------------
>
>                 Key: KAFKA-2871
>                 URL: https://issues.apache.org/jira/browse/KAFKA-2871
>             Project: Kafka
>          Issue Type: Bug
>          Components: replication
>    Affects Versions: 0.8.2.1
>            Reporter: Evan Huus
>            Assignee: Neha Narkhede
>            Priority: Minor
>
> We recently brought up a few brokers to replace some existing nodes, and used the provided script to reassign partitions from the retired nodes to the new ones, one at a time.
> A little while after the fact, we noticed extreme disk usage on the new nodes. Tracked this down to the fact that the replicated segments are all timestamped from the moment of replication rather than using whatever timestamp was set on the original node. Since this is the timestamp the log roller uses, it takes a full week (rollover time) before any data is purged from the new brokers.
> In the short term, what is the safest workaround? Can we just `rm` these old segments, or should we be messing with the filesystem metadata so kafka removes them itself?
> In the longer term, the partition mover should be setting timestamps appropriately on the segments it moves.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)