You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "aliehsaeedii (via GitHub)" <gi...@apache.org> on 2026/02/10 15:00:41 UTC

[PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

aliehsaeedii opened a new pull request, #21446:
URL: https://github.com/apache/kafka/pull/21446

   This PR add `RocksDBTimestampedStoreWithHeaders` and the corresponding untests (`RocksDBTimestampedStoreWithHeadersTest`) for the `TimestampedKeyValueStoreWithHeaders` introduced in [KIP-1271](https://cwiki.apache.org/confluence/display/KAFKA/KIP-1271%3A+Allow+to+Store+Record+Headers+in+State+Stores).


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2800959904


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");

Review Comment:
   the same



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");

Review Comment:
   the same



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801547618


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();

Review Comment:
   Yes, but there is only one record in the old CF -- and the iterator is supposed to correctly interleave between both CF, and return data across both in correct order. With only one record in the old CF, it seems to be not the best test coverage?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801552672


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");

Review Comment:
   I am talking about the `get` that we did earlier in the test: https://github.com/apache/kafka/pull/21446/changes#diff-5447f2dc1a76f9d433feadf7ac6d6c027d28f8b80e4edd1e96362922cdd6dfa9R136-R143
   
   Ie, this key should have been migrated before we start the iteration. -- Same for the other comment below. -- Or do I miss something?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #21446:
URL: https://github.com/apache/kafka/pull/21446


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2795833253


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */

Review Comment:
   If we rename the method as proposed, it seems we can remove this comment? The code is very clear that we open 3 CF anyway?



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();

Review Comment:
   Before we close the default CL, we should also check that it's empty -- if it's not empty, it could mean that the old store was actually a `RocksDBStore` (not `RocksDBTimestampStore`) and we don't support upgrading from non-ts-store to header-store directly; or, the old store was indeed a `RocksDBTimestampStore` but did not migrate all records to the "ts-CF" yet (and we would get data loss if we proceed).
   
   We can just throw an exception if default CL is not empty IMHO, to highlight the issue, and it's up to the user to do the right thing? Also need to add corresponding test.



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());

Review Comment:
   ```suggestion
           rocksDBStore.put(new Bytes("key8new".getBytes()), "headers+timestamp+88888888".getBytes());
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);

Review Comment:
   ```suggestion
               openInUpgradeMode(dbOptions, columnFamilyOptions);
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();
+
+        final ColumnFamilyHandle legacyCf = columnFamilies.get(1);
+        final ColumnFamilyHandle headersCf = columnFamilies.get(2);
+
+        // Check if legacy CF has data
+        final RocksIterator legacyIter = db.newIterator(legacyCf);
+        legacyIter.seekToFirst();
+        try {
+            if (legacyIter.isValid()) {
+                log.info("Opening store {} in upgrade mode", name);
+                cfAccessor = new DualColumnFamilyAccessor(legacyCf, headersCf,
+                    HeadersBytesStore::convertToHeaderFormat, this);
+            } else {
+                log.info("Opening store {} in regular headers-aware mode", name);

Review Comment:
   Should we also delete `legacyCf` for this case via `db.dropColumnFamily(...);` -- Otherwise, the code above would always be executed after an upgrade (even if he old CF is already empty)
   ```
           final boolean upgradingFromTimestampedStore = existingCFs.stream()
               .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
   ```
   
   If we do this, need to add a corresponding test.
   
   Side note: IIRC, it's not allowed to delete the default-CF, and that's the only reason why we did not do the delete for it in `RocksDBTimestampedStore`.



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();
+
+        final ColumnFamilyHandle legacyCf = columnFamilies.get(1);
+        final ColumnFamilyHandle headersCf = columnFamilies.get(2);
+
+        // Check if legacy CF has data
+        final RocksIterator legacyIter = db.newIterator(legacyCf);
+        legacyIter.seekToFirst();
+        try {
+            if (legacyIter.isValid()) {
+                log.info("Opening store {} in upgrade mode", name);
+                cfAccessor = new DualColumnFamilyAccessor(legacyCf, headersCf,
+                    HeadersBytesStore::convertToHeaderFormat, this);
+            } else {
+                log.info("Opening store {} in regular headers-aware mode", name);
+                cfAccessor = new SingleColumnFamilyAccessor(headersCf);
+                legacyCf.close();
+            }
+        } finally {
+            legacyIter.close();
+        }
+    }
+
+    /**
+     * Opens store with 2-CF design.
+     * CFs: DEFAULT (legacy timestamped without headers), headers (new)
+     */

Review Comment:
   Comments seems unnecessary -- it just says, that the code says by itself



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);

Review Comment:
   Might be good to explain where 22 comes from?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {

Review Comment:
   ```suggestion
       private void prepareTimestampedStore() {
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);

Review Comment:
   Should we make `RocksDBTimestampedStore#TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME` package private and re-use here? Either drop `LEGACY_TIMESTAMPED_CF_NAME` entirely, or define it as 
   ```
   private static final byte[] LEGACY_TIMESTAMPED_CF_NAME  = RocksDBTimestampedStore#TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME;
   ```
   
   This would also avoid the comment from above (and the need of the comment, is a tell)



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);

Review Comment:
   open
   ```suggestion
               openInRegularMode(dbOptions, columnFamilyOptions);
   ```



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {
+        // Create a legacy RocksDBTimestampedStore to test upgrade scenario
+        final RocksDBTimestampedStore timestampedStore = new RocksDBTimestampedStore(DB_NAME, METRICS_SCOPE);
+        try {
+            timestampedStore.init(context, timestampedStore);
+
+            // Write timestamped values (timestamp = -1 for unknown timestamp)

Review Comment:
   Why is `(timestamp = -1 for unknown timestamp)` relevant?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {
+        // Create a legacy RocksDBTimestampedStore to test upgrade scenario
+        final RocksDBTimestampedStore timestampedStore = new RocksDBTimestampedStore(DB_NAME, METRICS_SCOPE);
+        try {
+            timestampedStore.init(context, timestampedStore);
+
+            // Write timestamped values (timestamp = -1 for unknown timestamp)
+            timestampedStore.put(new Bytes("key1".getBytes()), wrapTimestampedValue("1".getBytes()));
+            timestampedStore.put(new Bytes("key2".getBytes()), wrapTimestampedValue("22".getBytes()));
+            timestampedStore.put(new Bytes("key3".getBytes()), wrapTimestampedValue("333".getBytes()));
+            timestampedStore.put(new Bytes("key4".getBytes()), wrapTimestampedValue("4444".getBytes()));
+            timestampedStore.put(new Bytes("key5".getBytes()), wrapTimestampedValue("55555".getBytes()));
+            timestampedStore.put(new Bytes("key6".getBytes()), wrapTimestampedValue("666666".getBytes()));
+            timestampedStore.put(new Bytes("key7".getBytes()), wrapTimestampedValue("7777777".getBytes()));
+        } finally {
+            timestampedStore.close();
+        }
+    }
+
+    private byte[] wrapTimestampedValue(final byte[] value) {

Review Comment:
   Why not pass in `long ts` as parameter, and create `new ValueAndTimestamp("" + ts, ts)` and use `ValueAndTimestampSerializer` to get the bytes?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");

Review Comment:
   Similar to above. `putIfAbsent` would not update the value, because the key exists, but still migrate the value from legacy- to headers-CF?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");

Review Comment:
   So for legacy CF count is already at zero, and thus stays at zero, even if we get the no-op delete?
   
   Should we add more row to the original store to avoid hitting count zero for better test coverage?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;

Review Comment:
   ```suggestion
           ColumnFamilyHandle defaultCF = null;
   ```



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());

Review Comment:
   ```suggestion
           rocksDBStore.put(new Bytes("key2".getBytes()), "headers+timestamp+22".getBytes());
   ```



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");

Review Comment:
   Same as for `key4`



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {

Review Comment:
   Why do we close these two only if there was an error?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");

Review Comment:
   Why does legacy count go down to `1` ? If we don't find a key, would we not do just one put into the header-CF -- or is there still one more no-op delete on legacy-CF and that's why the count is offset incorrectly by 1?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();

Review Comment:
   Seems, we only have `key7` in legacy CF when we enter this method? So not sure if we really get the signal we want to get?
   
   Guess we need to add more rows to the original store?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");

Review Comment:
   > from legacy CF
   
   Why? Wasn't key1 migrated on `get()`?



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.

Review Comment:
   This comment seems to not add any value -- I would remove it.



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);

Review Comment:
   ```suggestion
           assertEquals(1 + 0 + 8 + 12, db.get(headersColumnFamily, "key2".getBytes()).length);
   ```



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)

Review Comment:
   Should we use an `all()` iterator to verify that it's indeed empty? Point lookups seems not to do the trick?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {

Review Comment:
   We should also test with "plain" `RocksDBStore`, ie, test that we fail if uses try to upgrade from `RocksDBStore` to header-store. So we get two types of "old stores" so need more descriptive method name here.



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }

Review Comment:
   Should we also check that neither `key8` (aka `key8new`) nor`key11` are not in the legacy CF?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);

Review Comment:
   I am confuse why this test did not fail? We migrated `key2` to `timestamp+22` which is already 12 bytes, so it should be 21 bytes overall?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());

Review Comment:
   Doesn't really matter, but that's the format you use below, so might be good to be consistent?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;

Review Comment:
   Not sure why there is `2` at the end? Same for the other two below



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798674815


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {

Review Comment:
   I changed the code to make it more readable



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801552672


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");

Review Comment:
   I am talking about the `get` that we did earlier in the test: https://github.com/apache/kafka/pull/21446/changes#diff-5447f2dc1a76f9d433feadf7ac6d6c027d28f8b80e4edd1e96362922cdd6dfa9R136-R143
   
   Ie, this key should have been migrated before we start the iteration.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2817566606


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.hamcrest.core.IsNull;
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertThat(appender.getMessages(), hasItem("Opening store " + DB_NAME + " in regular headers-aware mode"));

Review Comment:
   Same for me in this case



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798423369


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();

Review Comment:
   Good catch!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798817801


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");

Review Comment:
   > there still one more no-op delete on legacy-CF 
   
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2803652938


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");

Review Comment:
   I reponded above.
   the iterator does not migrate any thing but does the conversion on the fly
   after iterating if when we call `verifyLegacyTimestampedColumnFamily` and `verifyHeadersColumnFamily` we see that nothing is migrated 



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801536341


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {
+        // Create a legacy RocksDBTimestampedStore to test upgrade scenario
+        final RocksDBTimestampedStore timestampedStore = new RocksDBTimestampedStore(DB_NAME, METRICS_SCOPE);
+        try {
+            timestampedStore.init(context, timestampedStore);
+
+            // Write timestamped values (timestamp = -1 for unknown timestamp)
+            timestampedStore.put(new Bytes("key1".getBytes()), wrapTimestampedValue("1".getBytes()));
+            timestampedStore.put(new Bytes("key2".getBytes()), wrapTimestampedValue("22".getBytes()));
+            timestampedStore.put(new Bytes("key3".getBytes()), wrapTimestampedValue("333".getBytes()));
+            timestampedStore.put(new Bytes("key4".getBytes()), wrapTimestampedValue("4444".getBytes()));
+            timestampedStore.put(new Bytes("key5".getBytes()), wrapTimestampedValue("55555".getBytes()));
+            timestampedStore.put(new Bytes("key6".getBytes()), wrapTimestampedValue("666666".getBytes()));
+            timestampedStore.put(new Bytes("key7".getBytes()), wrapTimestampedValue("7777777".getBytes()));
+        } finally {
+            timestampedStore.close();
+        }
+    }
+
+    private byte[] wrapTimestampedValue(final byte[] value) {

Review Comment:
   I see your point -- and in general I agree. But it can also be pushed to an unnecessary extreme; for this particular case, I think it would be totally ok to just use `ValueAndTimestamp` and `ValueAndTimestampSerializer` 🤷 
   
   In the end, if `ValueAndTimestampSerializer` has an issue, a _lot_ of tests (including most of our integration tests) would break anyway... And the fact that (hopefull) `ValueAndTimestampSerializerTest` breaks, tells us to start looking for the issue here :)
   
   Leave it up to you to make a decision either way.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2790449297


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final org.rocksdb.Options options = new org.rocksdb.Options(dbOptions, new ColumnFamilyOptions());

Review Comment:
   Could we replace full qualified name with import?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.hamcrest.core.IsNull;
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertThat(appender.getMessages(), hasItem("Opening store " + DB_NAME + " in regular headers-aware mode"));

Review Comment:
   I'm wondering should we keep using hamcrest in test.
   AFAIK, hamcrest is not active, maybe we should use JUnit instead?
   cc. @mjsax @bbejeck 



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final org.rocksdb.Options options = new org.rocksdb.Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final org.rocksdb.RocksDBException e) {
+            throw new org.apache.kafka.streams.errors.ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> java.util.Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));

Review Comment:
   ditto



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,141 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final org.rocksdb.Options options = new org.rocksdb.Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final org.rocksdb.RocksDBException e) {
+            throw new org.apache.kafka.streams.errors.ProcessorStateException("Error listing column families for store " + name, e);

Review Comment:
   ditto



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798750250


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);

Review Comment:
   I added the explanation, but we should know that the value is not longer than the pure value length as the put method did not add headers. Considering the purpose of the test, I think we can remove this check even.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2800952943


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");

Review Comment:
   // iterating should not migrate any data, but return all keys over both CFs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801124361


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);

Review Comment:
   No
   `key2` 's value was updated using `put` which inserts the given value directly to the value, but `key4` and `key5` got updated using `putIfAbsent` which calls `get` which means their values are converted to new format before being inserted in new CF.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798817801


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");

Review Comment:
   >there still one more no-op delete on legacy-CF 
   yes



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798380414


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);

Review Comment:
   Makes sense. Ensures consistency!



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2803654281


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");

Review Comment:
   iterator does the conversion on the fly



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2795800018


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,590 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.hamcrest.core.IsNull;
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.hamcrest.CoreMatchers.hasItem;
+import static org.hamcrest.CoreMatchers.is;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertThat(appender.getMessages(), hasItem("Opening store " + DB_NAME + " in regular headers-aware mode"));

Review Comment:
   I never pay close attention to this, and I am personally fine either way. Of course, if there is some overall effort to move off hamcrest in favor of JUnit, it would make sense to avoid adding hamcrest code.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798597454


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {
+        // Create a legacy RocksDBTimestampedStore to test upgrade scenario
+        final RocksDBTimestampedStore timestampedStore = new RocksDBTimestampedStore(DB_NAME, METRICS_SCOPE);
+        try {
+            timestampedStore.init(context, timestampedStore);
+
+            // Write timestamped values (timestamp = -1 for unknown timestamp)

Review Comment:
   Old comment. Removed! They used to be all -1, but I changed them.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2800959196


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();

Review Comment:
   // iterating should not migrate any data, but return all keys over both CFs



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2798607109


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key3".getBytes()));
+        assertEquals(1 + 0 + 8 + 4, db.get(headersColumnFamily, "key4".getBytes()).length);
+        assertEquals(1 + 0 + 8 + 5, db.get(headersColumnFamily, "key5".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key6".getBytes()));
+        assertNull(db.get(headersColumnFamily, "key7".getBytes()));
+        assertEquals(26, db.get(headersColumnFamily, "key8".getBytes()).length);
+        assertEquals(29, db.get(headersColumnFamily, "key11".getBytes()).length);
+        assertNull(db.get(headersColumnFamily, "key12".getBytes()));
+    }
+
+    private void closeColumnFamilies(
+            final RocksDB db,
+            final ColumnFamilyHandle defaultColumnFamily,
+            final ColumnFamilyHandle legacyTimestampedColumnFamily,
+            final ColumnFamilyHandle headersColumnFamily,
+            final DBOptions dbOptions,
+            final ColumnFamilyOptions columnFamilyOptions,
+            final boolean errorOccurred) {
+        // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+        if (defaultColumnFamily != null) {
+            defaultColumnFamily.close();
+        }
+        if (legacyTimestampedColumnFamily != null) {
+            legacyTimestampedColumnFamily.close();
+        }
+        if (headersColumnFamily != null) {
+            headersColumnFamily.close();
+        }
+        if (db != null) {
+            db.close();
+        }
+        if (errorOccurred) {
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyStillInUpgradeMode() {
+        // check that still in upgrade mode
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+    }
+
+    private void clearLegacyColumnFamily(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        // clear legacy timestamped CF by deleting remaining key
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultCF2 = null;
+        ColumnFamilyHandle legacyCF2 = null;
+        ColumnFamilyHandle headersCF2 = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultCF2 = columnFamilies.get(0);
+            legacyCF2 = columnFamilies.get(1);
+            headersCF2 = columnFamilies.get(2);
+            db.delete(legacyCF2, "key7".getBytes());
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultCF2 != null) {
+                defaultCF2.close();
+            }
+            if (legacyCF2 != null) {
+                legacyCF2.close();
+            }
+            if (headersCF2 != null) {
+                headersCF2.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    private void verifyInHeadersAwareMode() {
+        // check that now in regular header-aware mode (all legacy data migrated)
+        try (LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+    }
+
+    private void prepareOldStore() {
+        // Create a legacy RocksDBTimestampedStore to test upgrade scenario
+        final RocksDBTimestampedStore timestampedStore = new RocksDBTimestampedStore(DB_NAME, METRICS_SCOPE);
+        try {
+            timestampedStore.init(context, timestampedStore);
+
+            // Write timestamped values (timestamp = -1 for unknown timestamp)
+            timestampedStore.put(new Bytes("key1".getBytes()), wrapTimestampedValue("1".getBytes()));
+            timestampedStore.put(new Bytes("key2".getBytes()), wrapTimestampedValue("22".getBytes()));
+            timestampedStore.put(new Bytes("key3".getBytes()), wrapTimestampedValue("333".getBytes()));
+            timestampedStore.put(new Bytes("key4".getBytes()), wrapTimestampedValue("4444".getBytes()));
+            timestampedStore.put(new Bytes("key5".getBytes()), wrapTimestampedValue("55555".getBytes()));
+            timestampedStore.put(new Bytes("key6".getBytes()), wrapTimestampedValue("666666".getBytes()));
+            timestampedStore.put(new Bytes("key7".getBytes()), wrapTimestampedValue("7777777".getBytes()));
+        } finally {
+            timestampedStore.close();
+        }
+    }
+
+    private byte[] wrapTimestampedValue(final byte[] value) {

Review Comment:
   I can do that, but I was thinking of keeping the utests independent from other classes as much as possible. A mistake/change in  `ValueAndTimestampSerializer` should not affect rocksdb tests.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2806936701


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");

Review Comment:
   The idea was to get a better test signal. But maybe doesn't matter too much



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2800913582


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");

Review Comment:
   >Should we add more row to the original store to avoid hitting count zero for better test coverage?
   
   No, it's already a good coverage that shows this case as well. What would be difference of this (`key12`) with others then?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2803203088


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();

Review Comment:
   Ok, I left `key4` in the old CF as well to see the correct interleaving.
   BTW, `iteratorsShouldNotMigrateData` does not sattisfy our goal anyways as iterating should not migrate any data, but values from legacy CF are converted to header-aware format on the fly:  1 byte + [timestamp][value] which is the format of migrated data as well



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2803654281


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");

Review Comment:
   iterator does the conversion on the fly but does not apply it to the physical store.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801562469


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");
+
+        // delete() - tests migration on delete
+
+        assertEquals(1 + 0 + 8 + 6, rocksDBStore.delete(new Bytes("key6".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(6) = 15 bytes for delete() on existing key6");
+        assertEquals(3L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 3 in headers-aware CF after deleting key6 with delete()");
+
+        // iterators should not trigger migration (read-only)
+        iteratorsShouldNotMigrateData();
+        assertEquals(3L, rocksDBStore.approximateNumEntries());
+
+        rocksDBStore.close();
+
+        // Verify the final state of both column families
+        verifyOldAndNewColumnFamily();
+    }
+
+    private void iteratorsShouldNotMigrateData() {
+        // iterating should not migrate any data, but return all keys over both CFs
+        // Values from legacy CF are converted to header-aware format: [varint][headers][timestamp][value]
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.all()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.range(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> itAll = rocksDBStore.reverseAll()) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key8".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '8', '8', '8', '8', '8', '8', '8', '8'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key7".getBytes(), keyValue.key.get());
+                assertEquals(16, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(7) = 16 bytes for key7 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = itAll.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            assertFalse(itAll.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it =
+                          rocksDBStore.reverseRange(new Bytes("key2".getBytes()), new Bytes("key5".getBytes()))) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key5".getBytes(), keyValue.key.get());
+                assertEquals(14, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for key5 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key4".getBytes(), keyValue.key.get());
+                assertEquals(13, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(4) = 13 bytes for key4 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key2".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'t', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '2', '2'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> it = rocksDBStore.prefixScan("key1", stringSerializer)) {
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key1".getBytes(), keyValue.key.get());
+                assertEquals(10, keyValue.value.length, "Expected header-aware format: varint(0) + empty headers(0) + timestamp(8) + value(1) = 10 bytes for key1 from legacy CF");
+            }
+            {
+                final KeyValue<Bytes, byte[]> keyValue = it.next();
+                assertArrayEquals("key11".getBytes(), keyValue.key.get());
+                assertArrayEquals(new byte[]{'h', 'e', 'a', 'd', 'e', 'r', 's', '+', 't', 'i', 'm', 'e', 's', 't', 'a', 'm', 'p', '+', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1', '1'}, keyValue.value);
+            }
+            assertFalse(it.hasNext());
+        }
+    }
+
+    private void verifyOldAndNewColumnFamily() throws Exception {
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        // In upgrade scenario from RocksDBTimestampedStore,
+        // we expect 3 CFs: DEFAULT (closed on open), keyValueWithTimestamp (legacy), keyValueWithTimestampAndHeaders (new)
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8), columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+
+        verifyColumnFamilyContents(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyStillInUpgradeMode();
+        clearLegacyColumnFamily(dbOptions, columnFamilyDescriptors, columnFamilyOptions);
+        verifyInHeadersAwareMode();
+    }
+
+    private void verifyColumnFamilyContents(
+            final DBOptions dbOptions,
+            final List<ColumnFamilyDescriptor> columnFamilyDescriptors,
+            final ColumnFamilyOptions columnFamilyOptions) throws Exception {
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null;
+        ColumnFamilyHandle legacyTimestampedColumnFamily = null;
+        ColumnFamilyHandle headersColumnFamily = null;
+        boolean errorOccurred = false;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            legacyTimestampedColumnFamily = columnFamilies.get(1);
+            headersColumnFamily = columnFamilies.get(2);
+
+            verifyDefaultColumnFamily(db, defaultColumnFamily);
+            verifyLegacyTimestampedColumnFamily(db, legacyTimestampedColumnFamily);
+            verifyHeadersColumnFamily(db, headersColumnFamily);
+        } catch (final RuntimeException fatal) {
+            errorOccurred = true;
+        } finally {
+            closeColumnFamilies(db, defaultColumnFamily, legacyTimestampedColumnFamily, headersColumnFamily,
+                    dbOptions, columnFamilyOptions, errorOccurred);
+        }
+    }
+
+    private void verifyDefaultColumnFamily(final RocksDB db, final ColumnFamilyHandle defaultColumnFamily) throws Exception {
+        // DEFAULT CF should be empty (closed on open)
+        assertNull(db.get(defaultColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(defaultColumnFamily, "key1".getBytes()));
+    }
+
+    private void verifyLegacyTimestampedColumnFamily(final RocksDB db, final ColumnFamilyHandle legacyTimestampedColumnFamily) throws Exception {
+        // Legacy timestamped CF should have migrated keys as null, un-migrated as timestamped values
+        assertNull(db.get(legacyTimestampedColumnFamily, "unknown".getBytes()));
+        assertNull(db.get(legacyTimestampedColumnFamily, "key1".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key2".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key3".getBytes())); // deleted
+        assertNull(db.get(legacyTimestampedColumnFamily, "key4".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key5".getBytes())); // migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key6".getBytes())); // migrated
+        assertEquals(8 + 7, db.get(legacyTimestampedColumnFamily, "key7".getBytes()).length); // not migrated
+        assertNull(db.get(legacyTimestampedColumnFamily, "key8".getBytes()));
+    }
+
+    private void verifyHeadersColumnFamily(final RocksDB db, final ColumnFamilyHandle headersColumnFamily) throws Exception {
+        // Headers CF should have all migrated/new keys with header-aware format
+        assertNull(db.get(headersColumnFamily, "unknown".getBytes()));
+        assertEquals(1 + 0 + 8 + 1, db.get(headersColumnFamily, "key1".getBytes()).length); // varint + headers + ts + value
+        assertEquals(12, db.get(headersColumnFamily, "key2".getBytes()).length);

Review Comment:
   Oh I see... Because we are at the byte-store layer... (For the metered store layer, when we `put("timestamp+22")` it would get serialized and the serialization would add the header stuff...
   
   Worth to add a comment?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801543847


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");

Review Comment:
   It's just to avoid to drop down to `approximateNumEntries <= 0`. No need to actually "do something" with the newly added rows.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2801540737


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");

Review Comment:
   Thanks. Might be worth to add a comment about it.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2795850263


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();
+
+        final ColumnFamilyHandle legacyCf = columnFamilies.get(1);
+        final ColumnFamilyHandle headersCf = columnFamilies.get(2);
+
+        // Check if legacy CF has data
+        final RocksIterator legacyIter = db.newIterator(legacyCf);
+        legacyIter.seekToFirst();
+        try {
+            if (legacyIter.isValid()) {
+                log.info("Opening store {} in upgrade mode", name);
+                cfAccessor = new DualColumnFamilyAccessor(legacyCf, headersCf,
+                    HeadersBytesStore::convertToHeaderFormat, this);
+            } else {
+                log.info("Opening store {} in regular headers-aware mode", name);

Review Comment:
   Should we also delete `legacyCf` for this case via `db.dropColumnFamily(...);` -- Otherwise, the code above would always be executed after an upgrade (even if the old CF is already empty)
   ```
           final boolean upgradingFromTimestampedStore = existingCFs.stream()
               .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
   ```
   
   If we do this, need to add a corresponding test.
   
   Side note: IIRC, it's not allowed to delete the default-CF, and that's the only reason why we did not do the delete for it in `RocksDBTimestampedStore`.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2802895553


##########
streams/src/test/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeadersTest.java:
##########
@@ -0,0 +1,535 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogCaptureAppender;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.state.KeyValueIterator;
+
+import org.junit.jupiter.api.Test;
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+
+import java.io.File;
+import java.nio.charset.StandardCharsets;
+import java.util.ArrayList;
+import java.util.List;
+
+import static java.util.Arrays.asList;
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class RocksDBTimestampedStoreWithHeadersTest extends RocksDBStoreTest {
+
+    private final Serializer<String> stringSerializer = new StringSerializer();
+
+    RocksDBStore getRocksDBStore() {
+        return new RocksDBTimestampedStoreWithHeaders(DB_NAME, METRICS_SCOPE);
+    }
+
+    @Test
+    public void shouldOpenNewStoreInRegularMode() {
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        }
+
+        try (final KeyValueIterator<Bytes, byte[]> iterator = rocksDBStore.all()) {
+            assertFalse(iterator.hasNext());
+        }
+    }
+
+    @Test
+    public void shouldOpenExistingStoreInRegularMode() throws Exception {
+        // prepare store
+        rocksDBStore.init(context, rocksDBStore);
+        rocksDBStore.put(new Bytes("key".getBytes()), "timestampedWithHeaders".getBytes());
+        rocksDBStore.close();
+
+        // re-open store
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in regular headers-aware mode"));
+        } finally {
+            rocksDBStore.close();
+        }
+
+        // verify store
+        final DBOptions dbOptions = new DBOptions();
+        final ColumnFamilyOptions columnFamilyOptions = new ColumnFamilyOptions();
+
+        final List<ColumnFamilyDescriptor> columnFamilyDescriptors = asList(
+                new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+                new ColumnFamilyDescriptor("keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8), columnFamilyOptions));
+        final List<ColumnFamilyHandle> columnFamilies = new ArrayList<>(columnFamilyDescriptors.size());
+
+        RocksDB db = null;
+        ColumnFamilyHandle defaultColumnFamily = null, headersColumnFamily = null;
+        try {
+            db = RocksDB.open(
+                    dbOptions,
+                    new File(new File(context.stateDir(), "rocksdb"), DB_NAME).getAbsolutePath(),
+                    columnFamilyDescriptors,
+                    columnFamilies);
+
+            defaultColumnFamily = columnFamilies.get(0);
+            headersColumnFamily = columnFamilies.get(1);
+
+            assertNull(db.get(defaultColumnFamily, "key".getBytes()));
+            assertEquals(0L, db.getLongProperty(defaultColumnFamily, "rocksdb.estimate-num-keys"));
+            assertEquals(22, db.get(headersColumnFamily, "key".getBytes()).length);
+            assertEquals(1L, db.getLongProperty(headersColumnFamily, "rocksdb.estimate-num-keys"));
+        } finally {
+            // Order of closing must follow: ColumnFamilyHandle > RocksDB > DBOptions > ColumnFamilyOptions
+            if (defaultColumnFamily != null) {
+                defaultColumnFamily.close();
+            }
+            if (headersColumnFamily != null) {
+                headersColumnFamily.close();
+            }
+            if (db != null) {
+                db.close();
+            }
+            dbOptions.close();
+            columnFamilyOptions.close();
+        }
+    }
+
+    @Test
+    public void shouldMigrateFromTimestampedToHeadersAwareColumnFamily() throws Exception {
+        prepareOldStore();
+
+        // Open with RocksDBTimestampedStoreWithHeaders - should detect legacy CF and enter upgrade mode
+        try (final LogCaptureAppender appender = LogCaptureAppender.createAndRegister(RocksDBTimestampedStoreWithHeaders.class)) {
+            rocksDBStore.init(context, rocksDBStore);
+
+            assertTrue(appender.getMessages().contains("Opening store " + DB_NAME + " in upgrade mode"));
+        }
+
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries in legacy CF and 0 in headers-aware CF before migration");
+
+        // get() - tests lazy migration on read
+
+        assertNull(rocksDBStore.get(new Bytes("unknown".getBytes())), "Expected null for unknown key");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 7 entries on legacy CF, 0 in headers-aware CF");
+
+        assertEquals(1 + 0 + 8 + 1, rocksDBStore.get(new Bytes("key1".getBytes())).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(1) = 10 bytes");
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 6 entries on legacy CF, 1 in headers-aware CF after migrating key1");
+
+        // put() - tests migration on write
+
+        rocksDBStore.put(new Bytes("key2".getBytes()), "timestamp+22".getBytes());
+        assertEquals(7L, rocksDBStore.approximateNumEntries(), "Expected 5 entries on legacy CF, 2 in headers-aware CF after migrating key2 with put()");
+
+        rocksDBStore.put(new Bytes("key3".getBytes()), null);
+        // count is off by one, due to two delete operations (even if one does not delete anything)
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 4 entries on legacy CF, 1 in headers-aware CF after deleting key3 with put()");
+
+        rocksDBStore.put(new Bytes("key8".getBytes()), "headers+timestamp+88888888".getBytes());
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 3 entries on legacy CF, 2 in headers-aware CF after adding new key8 with put()");
+
+        // putIfAbsent() - tests migration on conditional write
+
+        assertEquals(1 + 0 + 8 + 4,
+            rocksDBStore.putIfAbsent(new Bytes("key4".getBytes()), "headers+timestamp+4444".getBytes()).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(4) = 13 bytes");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 2 entries on legacy CF, 3 in headers-aware CF after migrating key4 with putIfAbsent()");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key11".getBytes()), "headers+timestamp+11111111111".getBytes()),
+            "Expected null return value for putIfAbsent on non-existing key11, and new key should be added to headers-aware CF");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 1 entry on legacy CF, 4 in headers-aware CF after adding new key11 with putIfAbsent()");
+
+        assertEquals(1 + 0 + 8 + 5, rocksDBStore.putIfAbsent(new Bytes("key5".getBytes()), null).length,
+            "Expected header-aware format: varint(1) + empty headers(0) + timestamp(8) + value(5) = 14 bytes for putIfAbsent with null on existing key5");
+        assertEquals(5L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 5 in headers-aware CF after migrating key5 with putIfAbsent(null)");
+
+        assertNull(rocksDBStore.putIfAbsent(new Bytes("key12".getBytes()), null));
+        assertEquals(4L, rocksDBStore.approximateNumEntries(), "Expected 0 entries on legacy CF, 4 in headers-aware CF after putIfAbsent with null on non-existing key12");

Review Comment:
   >It's just to avoid to drop down to approximateNumEntries <= 0
   
   why should we avoid it? the test shows what happens when approximateNumEntries <= 0. I added a comment there as well.



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21446:
URL: https://github.com/apache/kafka/pull/21446#discussion_r2793049792


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }

Review Comment:
   ```suggestion
           try (final Options options = new Options(dbOptions, new ColumnFamilyOptions())) {
               existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
           } catch (final RocksDBException e) {
               throw new ProcessorStateException("Error listing column families for store " + name, e);
           }
   
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.
+    private static final byte[] TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME =
+        "keyValueWithTimestampAndHeaders".getBytes(StandardCharsets.UTF_8);
+
+    public RocksDBTimestampedStoreWithHeaders(final String name,
+                                              final String metricsScope) {
+        super(name, metricsScope);
+    }
+
+    RocksDBTimestampedStoreWithHeaders(final String name,
+                                       final String parentDir,
+                                       final RocksDBMetricsRecorder metricsRecorder) {
+        super(name, parentDir, metricsRecorder);
+    }
+
+    @Override
+    void openRocksDB(final DBOptions dbOptions,
+                     final ColumnFamilyOptions columnFamilyOptions) {
+        // Check if we're upgrading from RocksDBTimestampedStore (which uses keyValueWithTimestamp CF)
+        final List<byte[]> existingCFs;
+        try {
+            final Options options = new Options(dbOptions, new ColumnFamilyOptions());
+            existingCFs = RocksDB.listColumnFamilies(options, dbDir.getAbsolutePath());
+            options.close();
+        } catch (final RocksDBException e) {
+            throw new ProcessorStateException("Error listing column families for store " + name, e);
+        }
+
+        final boolean upgradingFromTimestampedStore = existingCFs.stream()
+            .anyMatch(cf -> Arrays.equals(cf, LEGACY_TIMESTAMPED_CF_NAME));
+
+        if (upgradingFromTimestampedStore) {
+            openWithLegacyTimestampedCF(dbOptions, columnFamilyOptions);
+        } else {
+            openWith2CFs(dbOptions, columnFamilyOptions);
+        }
+    }
+
+    /**
+     * Opens store when upgrading from RocksDBTimestampedStore (3 CFs).
+     * CFs: DEFAULT (closed), keyValueWithTimestamp (legacy), headers (new)
+     */
+    private void openWithLegacyTimestampedCF(final DBOptions dbOptions,
+                                              final ColumnFamilyOptions columnFamilyOptions) {
+        final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+            dbOptions,
+            // we have to open the default CF to be able to open the legacy CF, but we won't use it
+            new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+            new ColumnFamilyDescriptor(LEGACY_TIMESTAMPED_CF_NAME, columnFamilyOptions),
+            new ColumnFamilyDescriptor(TIMESTAMPED_VALUES_WITH_HEADERS_CF_NAME, columnFamilyOptions)
+        );
+        // Close unused default CF
+        columnFamilies.get(0).close();
+
+        final ColumnFamilyHandle legacyCf = columnFamilies.get(1);
+        final ColumnFamilyHandle headersCf = columnFamilies.get(2);
+
+        // Check if legacy CF has data
+        final RocksIterator legacyIter = db.newIterator(legacyCf);
+        legacyIter.seekToFirst();
+        try {
+            if (legacyIter.isValid()) {
+                log.info("Opening store {} in upgrade mode", name);
+                cfAccessor = new DualColumnFamilyAccessor(legacyCf, headersCf,
+                    HeadersBytesStore::convertToHeaderFormat, this);
+            } else {
+                log.info("Opening store {} in regular headers-aware mode", name);
+                cfAccessor = new SingleColumnFamilyAccessor(headersCf);
+                legacyCf.close();
+            }
+        } finally {
+            legacyIter.close();
+        }

Review Comment:
   ```suggestion
           try (final RocksIterator legacyIter = db.newIterator(legacyCf)) {
               legacyIter.seekToFirst();
               if (legacyIter.isValid()) {
                   log.info("Opening store {} in upgrade mode", name);
                   cfAccessor = new DualColumnFamilyAccessor(legacyCf, headersCf,
                       HeadersBytesStore::convertToHeaderFormat, this);
               } else {
                   log.info("Opening store {} in regular headers-aware mode", name);
                   cfAccessor = new SingleColumnFamilyAccessor(headersCf);
                   legacyCf.close();
               }
           } 
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME
+    private static final byte[] LEGACY_TIMESTAMPED_CF_NAME =
+        "keyValueWithTimestamp".getBytes(StandardCharsets.UTF_8);
+
+    // New column family for header-aware timestamped values.

Review Comment:
   ```suggestion
       /**
        * New column family for header-aware timestamped values.
        */
   ```



##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimestampedStoreWithHeaders.java:
##########
@@ -0,0 +1,145 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.Options;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksDBException;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.charset.StandardCharsets;
+import java.util.Arrays;
+import java.util.List;
+
+/**
+ * A persistent key-(value-timestamp-headers) store based on RocksDB.
+ */
+public class RocksDBTimestampedStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+
+    private static final Logger log = LoggerFactory.getLogger(RocksDBTimestampedStoreWithHeaders.class);
+
+    // Legacy column family name - must match RocksDBTimestampedStore.TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME

Review Comment:
   ```suggestion
       /**
        * Legacy column family name - must match {@code RocksDBTimestampedStore#TIMESTAMPED_VALUES_COLUMN_FAMILY_NAME} 
        */
   
   ```



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


Re: [PR] KAFKA-20132: Implement TimestampedKeyValueStoreWithHeaders (2/N) [kafka]

Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21446:
URL: https://github.com/apache/kafka/pull/21446#issuecomment-3887921295

   Please rebase to `trunk` to ensure we don't miss anything about the checkstyle/spotbugs fix.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org