[jira] [Created] (KAFKA-15682) Ensure internal remote log metadata topic does not expire its segments before deleting user-topic segments

["Kamal Chandraprakash (Jira)" <ji...@apache.org>]

Kamal Chandraprakash created KAFKA-15682:
--------------------------------------------

             Summary: Ensure internal remote log metadata topic does not expire its segments before deleting user-topic segments
                 Key: KAFKA-15682
                 URL: https://issues.apache.org/jira/browse/KAFKA-15682
             Project: Kafka
          Issue Type: Task
            Reporter: Kamal Chandraprakash


One of the implementation of RemoteLogMetadataManager is TopicBasedRemoteLogMetadataManager which uses an internal Kafka topic. Unlike other internal topics which are compaction enabled, this topic is not enabled with compaction and retention is set to unlimited. 

Keeping this internal topic retention to unlimited is not practical in real world use-case where the topic disk usage footprint grows large over a period of time. 

It is assumed that the user will set the retention to a reasonable time such that it is the max of all the user-created topics (max + X). We can't just rely on it and need an assertion before deleting the internal {{__remote_log_metadata}} segments, otherwise there will be dangling remote log segments which won't be cleared once all the brokers are restarted post the topic truncation.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)