You are viewing a plain text version of this content. The canonical link for it is here.

Posted to jira@kafka.apache.org by "Michael Axiak (JIRA)" <ji...@apache.org> on 2019/05/20 15:39:00 UTC

[jira] [Created] (KAFKA-8395) Add an ability to backup log segment files on truncation

Michael Axiak created KAFKA-8395:
------------------------------------

             Summary: Add an ability to backup log segment files on truncation
                 Key: KAFKA-8395
                 URL: https://issues.apache.org/jira/browse/KAFKA-8395
             Project: Kafka
          Issue Type: New Feature
          Components: core
    Affects Versions: 2.2.0
            Reporter: Michael Axiak
             Fix For: 2.2.1


At HubSpot, we believe we hit a combination of bugs [1] [2], which may have caused us to lose data. In this scenario, as part of metadata conflict resolution a slowly starting up broker recovered an offset of zero and truncated segment files.

As part of a belt-and-suspenders approach to reducing this risk in the future, I propose adding the ability to rename/backup these files and allowing kafka to move on. Note that this breaks the ordering guarantees, but allows one to recover the data and decide later how to approach it.

This feature should be turned off by default but enabled with a configuration option.

(A pull request is following soon on Github)

1: https://issues.apache.org/jira/browse/KAFKA-2178
 2: https://issues.apache.org/jira/browse/KAFKA-1120

 



--
This message was sent by Atlassian JIRA
(v7.6.3#76005)