[jira] [Created] (KAFKA-15195) Regenerate segment-aligned producer snapshots when upgrading to a Kafka version supporting Tiered Storage

["Christo Lolov (Jira)" <ji...@apache.org>]

Christo Lolov created KAFKA-15195:
-------------------------------------

             Summary: Regenerate segment-aligned producer snapshots when upgrading to a Kafka version supporting Tiered Storage
                 Key: KAFKA-15195
                 URL: https://issues.apache.org/jira/browse/KAFKA-15195
             Project: Kafka
          Issue Type: Sub-task
    Affects Versions: 3.6.0
            Reporter: Christo Lolov
            Assignee: Christo Lolov


As mentioned in KIP-405: Kafka Tiered Storage#Upgrade a customer wishing to upgrade from a Kafka version < 2.8.0 to 3.6 and turn Tiered Storage on will have to wait for retention to clean up segments without an associated producer snapshot.

However, in our experience, customers of Kafka expect to be able to immediately enable tiering on a topic once their cluster upgrade is complete. Once they do this, however, they start seeing NPEs and no data is uploaded to Tiered Storage (https://github.com/apache/kafka/blob/9e50f7cdd37f923cfef4711cf11c1c5271a0a6c7/storage/api/src/main/java/org/apache/kafka/server/log/remote/storage/LogSegmentData.java#L61).

To achieve this, we propose changing Kafka to retroactively create producer snapshot files on upload whenever a segment is due to be archived and lacks one.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)