[jira] [Created] (KAFKA-5425) Kafka replication support all log segment files sync with leader

["Pengwei (JIRA)" <ji...@apache.org>]

Pengwei created KAFKA-5425:
------------------------------

             Summary: Kafka replication support all log segment files sync with leader
                 Key: KAFKA-5425
                 URL: https://issues.apache.org/jira/browse/KAFKA-5425
             Project: Kafka
          Issue Type: Improvement
            Reporter: Pengwei
            Priority: Minor


Currently kafka replication only support follower sync the leader with the 
latest  fetch offset, it means offsets smaller than this fetch offset's messages are already sync.  After complete sync with the leader, it will update the fetch offset to log end offset.

But if the log segments which latest offset is smaller than current fetch offset are being delete, the replication will not sync these log segments back to kafka.   In this case, there is a risk of losing messages if the follow is becoming a leader,  consumer begin to consume from begin to end will found some messages are lost.

Can we  improve the kafka replication mechanism to ensure follow''s all log segments are the same as the leader? Even if some log segments are delete,  will try to recover from the leader



--
This message was sent by Atlassian JIRA
(v6.3.15#6346)