You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Bill Bejeck (Jira)" <ji...@apache.org> on 2026/04/29 15:00:00 UTC
[jira] [Updated] (KAFKA-20543) Drop RocksDB offsets column family on clean shutdown when downgrading from 4.3
[ https://issues.apache.org/jira/browse/KAFKA-20543?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Bill Bejeck updated KAFKA-20543:
--------------------------------
Description:
KIP-1035 introduced a dedicated offsets column family in each RocksDB instance to persist changelog offsets per-store. KAFKA-19710 added a downgrade path that writes a legacy per-task .checkpoint file on clean shutdown when upgrade.from is set to a pre-4.3 version, but it does not remove the offsets column family from RocksDB. As a result, older Kafka Streams versions (which do not declare this column family at open time) fail with RocksDBException and the application crashes on startup, making in-flight downgrade impossible.
This ticket covers extending the clean-shutdown downgrade path to also drop the offsets column family (via db.dropColumnFamily(handle)) in RocksDBStore, RocksDBTimestampedStore, RocksDBTimestampedStoreWithHeaders, RocksDBMigratingWindowStoreWithHeaders, and RocksDBMigratingSessionStoreWithHeaders before the underlying RocksDB instance is closed, so that older versions can open the store and resume from the legacy .checkpoint file written by KAFKA-19710. Once this is in place, the downgrade documentation in docs/streams/upgrade-guide.md should be updated to describe the supported in-flight upgrade.from procedure instead of the current wipe-and-restore workaround.
was:
KIP-1035 introduced a dedicated offsets column family in each RocksDB instance to persist changelog offsets per-store. KAFKA-19710 added a downgrade path that writes a legacy per-task .checkpoint file on
clean shutdown when upgrade.from is set to a pre-4.3 version, but it does not remove the offsets column family from RocksDB. As a result, older Kafka Streams versions (which do not declare this column
family at open time) fail with RocksDBException and the application crashes on startup, making in-flight downgrade impossible.
This ticket covers extending the clean-shutdown downgrade path to also drop the offsets column family (via db.dropColumnFamily(handle)) in RocksDBStore, RocksDBTimestampedStore,
RocksDBTimestampedStoreWithHeaders, RocksDBMigratingWindowStoreWithHeaders, and RocksDBMigratingSessionStoreWithHeaders before the underlying RocksDB instance is closed, so that older versions can open the
store and resume from the legacy .checkpoint file written by KAFKA-19710. Once this is in place, the downgrade documentation in docs/streams/upgrade-guide.md should be updated to describe the supported
in-flight upgrade.from procedure instead of the current wipe-and-restore workaround.
> Drop RocksDB offsets column family on clean shutdown when downgrading from 4.3
> -------------------------------------------------------------------------------
>
> Key: KAFKA-20543
> URL: https://issues.apache.org/jira/browse/KAFKA-20543
> Project: Kafka
> Issue Type: Improvement
> Components: streams
> Affects Versions: 4.3.0
> Reporter: Bill Bejeck
> Assignee: Bill Bejeck
> Priority: Major
> Fix For: 4.3.1, 4.4.0
>
>
> KIP-1035 introduced a dedicated offsets column family in each RocksDB instance to persist changelog offsets per-store. KAFKA-19710 added a downgrade path that writes a legacy per-task .checkpoint file on clean shutdown when upgrade.from is set to a pre-4.3 version, but it does not remove the offsets column family from RocksDB. As a result, older Kafka Streams versions (which do not declare this column family at open time) fail with RocksDBException and the application crashes on startup, making in-flight downgrade impossible.
>
> This ticket covers extending the clean-shutdown downgrade path to also drop the offsets column family (via db.dropColumnFamily(handle)) in RocksDBStore, RocksDBTimestampedStore, RocksDBTimestampedStoreWithHeaders, RocksDBMigratingWindowStoreWithHeaders, and RocksDBMigratingSessionStoreWithHeaders before the underlying RocksDB instance is closed, so that older versions can open the store and resume from the legacy .checkpoint file written by KAFKA-19710. Once this is in place, the downgrade documentation in docs/streams/upgrade-guide.md should be updated to describe the supported in-flight upgrade.from procedure instead of the current wipe-and-restore workaround.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)