[jira] [Created] (KAFKA-20543) Drop RocksDB offsets column family on clean shutdown when downgrading from 4.3

["Bill Bejeck (Jira)" <ji...@apache.org>]

Bill Bejeck created KAFKA-20543:
-----------------------------------

             Summary: Drop RocksDB offsets column family on clean shutdown when downgrading from 4.3 
                 Key: KAFKA-20543
                 URL: https://issues.apache.org/jira/browse/KAFKA-20543
             Project: Kafka
          Issue Type: Improvement
          Components: streams
    Affects Versions: 4.3.0
            Reporter: Bill Bejeck
            Assignee: Bill Bejeck
             Fix For: 4.3.1, 4.4.0


KIP-1035 introduced a dedicated offsets column family in each RocksDB instance to persist changelog offsets per-store. KAFKA-19710 added a downgrade path that writes a legacy per-task .checkpoint file on  
  clean shutdown when upgrade.from is set to a pre-4.3 version, but it does not remove the offsets column family from RocksDB. As a result, older Kafka Streams versions (which do not declare this column
  family at open time) fail with RocksDBException and the application crashes on startup, making in-flight downgrade impossible.                                                                               
                                                            
  This ticket covers extending the clean-shutdown downgrade path to also drop the offsets column family (via db.dropColumnFamily(handle)) in RocksDBStore, RocksDBTimestampedStore,                            
  RocksDBTimestampedStoreWithHeaders, RocksDBMigratingWindowStoreWithHeaders, and RocksDBMigratingSessionStoreWithHeaders before the underlying RocksDB instance is closed, so that older versions can open the
   store and resume from the legacy .checkpoint file written by KAFKA-19710. Once this is in place, the downgrade documentation in docs/streams/upgrade-guide.md should be updated to describe the supported   
  in-flight upgrade.from procedure instead of the current wipe-and-restore workaround.



--
This message was sent by Atlassian Jira
(v8.20.10#820010)