You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2026/03/16 20:56:19 UTC
[PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
frankvicky opened a new pull request, #21780:
URL: https://github.com/apache/kafka/pull/21780
This PR enables time-ordered window store with headers.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "bbejeck (via GitHub)" <gi...@apache.org>.
bbejeck commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2943729807
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBMigratingWindowStoreWithHeaders.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-value store with headers support based on RocksDB for window stores.
+ * <p>
+ * This store provides a migration path from plain {@link RocksDBStore} (DEFAULT column family)
+ * to a headers-aware column family ({@code windowKeyValueWithHeaders}). It uses
+ * {@link DualColumnFamilyAccessor} for lazy migration when legacy data exists in the DEFAULT CF.
+ * <p>
+ * Value format:
+ * <ul>
+ * <li>Old format (DEFAULT CF): {@code [value]}</li>
+ * <li>New format (HEADERS CF): {@code [headersSize(varint)][headersBytes][value]}</li>
+ * </ul>
+ * <p>
+ * This is similar to {@link RocksDBMigratingSessionStoreWithHeaders} but for window stores.
+ * Unlike timestamped stores, window stores don't include timestamp in the value (it's in the key).
+ */
+public class RocksDBMigratingWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+ private static final Logger log = LoggerFactory.getLogger(RocksDBMigratingWindowStoreWithHeaders.class);
+
+ static final byte[] WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME =
+ "windowKeyValueWithHeaders".getBytes(StandardCharsets.UTF_8);
+
+ RocksDBMigratingWindowStoreWithHeaders(final String name,
+ final String parentDir,
+ final RocksDBMetricsRecorder metricsRecorder) {
+ super(name, parentDir, metricsRecorder);
+ }
+
+ @Override
+ void openRocksDB(final DBOptions dbOptions,
+ final ColumnFamilyOptions columnFamilyOptions) {
+ final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+ dbOptions,
+ new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+ new ColumnFamilyDescriptor(WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME, columnFamilyOptions),
+ new ColumnFamilyDescriptor(OFFSETS_COLUMN_FAMILY_NAME, columnFamilyOptions)
+ );
+ final ColumnFamilyHandle noHeadersColumnFamily = columnFamilies.get(0);
+ final ColumnFamilyHandle withHeadersColumnFamily = columnFamilies.get(1);
+ final ColumnFamilyHandle offsetsCf = columnFamilies.get(2);
+
+ // Check if DEFAULT CF has data (upgrade from old format without headers)
+ try (final RocksIterator noHeadersIter = db.newIterator(noHeadersColumnFamily)) {
+ noHeadersIter.seekToFirst();
+ if (noHeadersIter.isValid()) {
+ log.info("Opening window store {} in upgrade mode from plain value format", name);
+ // Migrate from [value] to [headers][value]
+ // Add empty headers prefix [0x00] to plain value
+ cfAccessor = new DualColumnFamilyAccessor(
+ offsetsCf,
+ noHeadersColumnFamily,
+ withHeadersColumnFamily,
+ RocksDBMigratingWindowStoreWithHeaders::addEmptyHeadersPrefix,
+ this,
+ open
+ );
+ } else {
+ log.info("Opening window store {} in regular headers-aware mode", name);
+ cfAccessor = new SingleColumnFamilyAccessor(offsetsCf, withHeadersColumnFamily);
+ noHeadersColumnFamily.close();
+ }
+ }
+ }
+
+ /**
+ * Converts plain value to value with empty headers prefix.
+ * <p>
+ * For window stores, the legacy format is: [value]
+ * The new format is: [headersSize(varint)][headersBytes][value]
+ * <p>
+ * This method adds empty headers prefix to the plain value.
+ * Empty headers: headersSize = 0 (encoded as [0x00])
+ *
+ * @param plainValue the plain value bytes
+ * @return the value with empty headers prefix: [0x00][plainValue]
+ */
+ private static byte[] addEmptyHeadersPrefix(final byte[] plainValue) {
+ if (plainValue == null) {
+ return null;
+ }
+ // For empty headers: headersSize = varint(0) = [0x00]
+ return ByteBuffer.allocate(1 + plainValue.length)
Review Comment:
Maybe use `System.arraycopy` here instead?
Actually this is virtually the same as `HeadersBytesStore.convertToHeaderFormat` but probably semantically more correct to use this version since it's a plain value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2966317810
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
Review Comment:
I suppose we don't need to?
The `retainDuplicates` parameter only affects the seqnum in the RocksDB key, not the headers serialization in the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2957568109
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStore.java:
##########
@@ -36,7 +36,7 @@
public class RocksDBTimeOrderedWindowStore
- extends WrappedStateStore<RocksDBTimeOrderedWindowSegmentedBytesStore, Object, Object>
+ extends WrappedStateStore<AbstractRocksDBTimeOrderedSegmentedBytesStore<? extends Segment>, Object, Object>
Review Comment:
I included a fix for this in https://github.com/apache/kafka/pull/21794 already
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2955497253
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ final HeadersSerializer.PreSerializedHeaders preSerializedHeaders = HeadersSerializer.prepareSerialization(headers);
+ final byte[] rawHeaders = HeadersSerializer.serialize(
+ preSerializedHeaders,
+ ByteBuffer.allocate(preSerializedHeaders.requiredBufferSizeForHeaders)
+ ).array();
+
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ ByteUtils.writeVarint(rawHeaders.length, out);
+ out.write(rawHeaders);
+ out.write(value);
+ return baos.toByteArray();
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to serialize value with headers", e);
+ }
+ }
+
+ /**
+ * Helper method to extract raw value from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] extractRawValue(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+ return rawValue;
+ }
+
+ /**
+ * Helper method to extract headers from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static Headers extractHeaders(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ final byte[] headerBytes = new byte[headersSize];
+ buffer.get(headerBytes);
+ return HeadersDeserializer.deserialize(headerBytes);
+ }
+
+ @Test
+ public void shouldMigrateFromWithoutHeadersToWithHeaders() {
+ final RocksDbIndexedTimeOrderedWindowBytesStoreSupplier oldSupplier =
+ new RocksDbIndexedTimeOrderedWindowBytesStoreSupplier(
+ STORE_NAME,
+ RETENTION_MS,
+ SEGMENT_INTERVAL_MS,
+ WINDOW_SIZE_MS,
+ false,
+ true,
Review Comment:
Should `withIndex` be `false` here? There is `shouldMigrateFromWithIndexToWithIndexAndHeaders` below.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
Review Comment:
It seems all tests use `retainDuplicates == false` -- is this sufficient? Not sure from top of my head if we would need to also test with `retainDuplicates == true`. Maybe not?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
Review Comment:
Thinking about this, we might want to rename `AggregationWithHeadersSerializer` (and related classes) to `ValueWithHeadersSerializer` to make it more generic? We introduced it for session-store for which it's indeed always the session-aggregation we store, but now with the new store type we introduce in this PR, we can re-use it, as it's more general purpose now?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ final HeadersSerializer.PreSerializedHeaders preSerializedHeaders = HeadersSerializer.prepareSerialization(headers);
+ final byte[] rawHeaders = HeadersSerializer.serialize(
+ preSerializedHeaders,
+ ByteBuffer.allocate(preSerializedHeaders.requiredBufferSizeForHeaders)
+ ).array();
+
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ ByteUtils.writeVarint(rawHeaders.length, out);
+ out.write(rawHeaders);
+ out.write(value);
+ return baos.toByteArray();
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to serialize value with headers", e);
+ }
+ }
+
+ /**
+ * Helper method to extract raw value from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] extractRawValue(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+ return rawValue;
+ }
+
+ /**
+ * Helper method to extract headers from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static Headers extractHeaders(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ final byte[] headerBytes = new byte[headersSize];
+ buffer.get(headerBytes);
+ return HeadersDeserializer.deserialize(headerBytes);
+ }
+
+ @Test
+ public void shouldMigrateFromWithoutHeadersToWithHeaders() {
+ final RocksDbIndexedTimeOrderedWindowBytesStoreSupplier oldSupplier =
+ new RocksDbIndexedTimeOrderedWindowBytesStoreSupplier(
+ STORE_NAME,
+ RETENTION_MS,
+ SEGMENT_INTERVAL_MS,
+ WINDOW_SIZE_MS,
+ false,
+ true,
Review Comment:
Well, I also see that there is `shouldMigrateFromWithoutIndexToWithIndexAndHeaders` even further below.
Should we just delete the whole test -- seems it got duplicated?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ final HeadersSerializer.PreSerializedHeaders preSerializedHeaders = HeadersSerializer.prepareSerialization(headers);
+ final byte[] rawHeaders = HeadersSerializer.serialize(
+ preSerializedHeaders,
+ ByteBuffer.allocate(preSerializedHeaders.requiredBufferSizeForHeaders)
+ ).array();
+
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ ByteUtils.writeVarint(rawHeaders.length, out);
+ out.write(rawHeaders);
+ out.write(value);
+ return baos.toByteArray();
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to serialize value with headers", e);
+ }
+ }
+
+ /**
+ * Helper method to extract raw value from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] extractRawValue(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+ return rawValue;
+ }
+
+ /**
+ * Helper method to extract headers from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static Headers extractHeaders(final byte[] valueWithHeaders) {
Review Comment:
Same question.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ final HeadersSerializer.PreSerializedHeaders preSerializedHeaders = HeadersSerializer.prepareSerialization(headers);
+ final byte[] rawHeaders = HeadersSerializer.serialize(
+ preSerializedHeaders,
+ ByteBuffer.allocate(preSerializedHeaders.requiredBufferSizeForHeaders)
+ ).array();
+
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ ByteUtils.writeVarint(rawHeaders.length, out);
+ out.write(rawHeaders);
+ out.write(value);
+ return baos.toByteArray();
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to serialize value with headers", e);
+ }
+ }
+
+ /**
+ * Helper method to extract raw value from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] extractRawValue(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+ return rawValue;
+ }
+
+ /**
+ * Helper method to extract headers from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static Headers extractHeaders(final byte[] valueWithHeaders) {
+ if (valueWithHeaders == null) {
+ return null;
+ }
+ final ByteBuffer buffer = ByteBuffer.wrap(valueWithHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ final byte[] headerBytes = new byte[headersSize];
+ buffer.get(headerBytes);
+ return HeadersDeserializer.deserialize(headerBytes);
+ }
+
+ @Test
+ public void shouldMigrateFromWithoutHeadersToWithHeaders() {
+ final RocksDbIndexedTimeOrderedWindowBytesStoreSupplier oldSupplier =
+ new RocksDbIndexedTimeOrderedWindowBytesStoreSupplier(
+ STORE_NAME,
+ RETENTION_MS,
+ SEGMENT_INTERVAL_MS,
+ WINDOW_SIZE_MS,
+ false,
+ true,
+ false // withHeaders = FALSE (old format)
+ );
+
+ final WindowStore<Bytes, byte[]> oldStore = oldSupplier.get();
+ oldStore.init(context, oldStore);
+
+ // Write data in old format
+ final long baseTime = System.currentTimeMillis();
+ final Bytes key1 = Bytes.wrap("key1".getBytes());
+ final Bytes key2 = Bytes.wrap("key2".getBytes());
+ final Bytes key3 = Bytes.wrap("key3".getBytes());
+
+ oldStore.put(key1, "value1".getBytes(), baseTime + 100);
+ oldStore.put(key2, "value2".getBytes(), baseTime + 200);
+ oldStore.put(key3, "value3".getBytes(), baseTime + 300);
+
+ // Verify old data
+ assertEquals("value1", new String(oldStore.fetch(key1, baseTime + 100)));
+ assertEquals("value2", new String(oldStore.fetch(key2, baseTime + 200)));
+ assertEquals("value3", new String(oldStore.fetch(key3, baseTime + 300)));
+
+ oldStore.close();
+
+ final RocksDbIndexedTimeOrderedWindowBytesStoreSupplier newSupplier =
+ new RocksDbIndexedTimeOrderedWindowBytesStoreSupplier(
+ STORE_NAME,
+ RETENTION_MS,
+ SEGMENT_INTERVAL_MS,
+ WINDOW_SIZE_MS,
+ false, // retainDuplicates
+ true, // withIndex
Review Comment:
Should `withIndex` be `false` here?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStore.java:
##########
@@ -36,7 +36,7 @@
public class RocksDBTimeOrderedWindowStore
- extends WrappedStateStore<RocksDBTimeOrderedWindowSegmentedBytesStore, Object, Object>
+ extends WrappedStateStore<AbstractRocksDBTimeOrderedSegmentedBytesStore<? extends Segment>, Object, Object>
Review Comment:
Should we add a generic type to `RocksDBTimeOrderedWindowStore<S extends Segment>` and use `AbstractRocksDBTimeOrderedSegmentedBytesStore<S>` here?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStoreWithHeaders.java:
##########
@@ -26,25 +31,41 @@
* {@link TimestampedBytesStore} (for timestamp support) and {@link HeadersBytesStore}
* (for header support) marker interfaces.
* <p>
- * The storage format for values is: [headersSize(varint)][headersBytes][timestamp(8)][value]
+ * This store returns {@link QueryResult#forUnknownQueryType(Query, Object)} for all queries,
+ * as IQv2 query handling is done at the metered layer.
* <p>
- * This implementation uses segment-level versioning for backward compatibility:
- * <ul>
- * <li>Old segments continue to use the legacy format without headers</li>
- * <li>New segments use the header-embedded format</li>
- * <li>Legacy values are served with empty headers on read</li>
- * <li>All new writes use the new format</li>
- * </ul>
+ * The storage format for values is: [headersSize(varint)][headersBytes][timestamp(8)][value]
*
* @see RocksDBTimeOrderedWindowStore
* @see HeadersBytesStore
* @see TimestampedBytesStore
*/
class RocksDBTimeOrderedWindowStoreWithHeaders extends RocksDBTimeOrderedWindowStore implements TimestampedBytesStore, HeadersBytesStore {
Review Comment:
Follow up to my above question. Should this be `extends RocksDBTimeOrderedWindowStore<WindowSegmentWithHeaders>` so we tie up the types correctly?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
Review Comment:
I think we don't need this helper? It does the exact same thing as `AggregationWithHeadersSerializer` I believe?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/TimeOrderedWindowStoreUpgradeTest.java:
##########
@@ -0,0 +1,301 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.StreamsTestUtils;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.File;
+import java.io.IOException;
+import java.nio.ByteBuffer;
+import java.time.Duration;
+import java.util.Properties;
+
+import static org.apache.kafka.common.utils.Utils.delete;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+/**
+ * Tests upgrade path from time-ordered window store (without headers) to
+ * time-ordered window store with headers using direct supplier creation.
+ * <p>
+ * This test validates lazy migration from ts-CF to headers-CF and ensures
+ * the store can read old data after upgrade.
+ */
+public class TimeOrderedWindowStoreUpgradeTest {
+
+ private static final String STORE_NAME = "time-ordered-window-store";
+ private static final long WINDOW_SIZE_MS = 1000L;
+ private static final long RETENTION_MS = Duration.ofMinutes(5).toMillis();
+ private static final long SEGMENT_INTERVAL_MS = Duration.ofMinutes(1).toMillis();
+
+ private InternalMockProcessorContext<Bytes, byte[]> context;
+ private File baseDir;
+
+ @BeforeEach
+ public void setUp() {
+ final Properties props = StreamsTestUtils.getStreamsConfig();
+ baseDir = TestUtils.tempDirectory();
+ context = new InternalMockProcessorContext<>(
+ baseDir,
+ Serdes.Bytes(),
+ Serdes.ByteArray(),
+ new StreamsConfig(props)
+ );
+ }
+
+ @AfterEach
+ public void tearDown() {
+ if (baseDir != null) {
+ try {
+ delete(baseDir);
+ } catch (final Exception e) {
+ // Ignore
+ }
+ }
+ }
+
+ /**
+ * Helper method to serialize value with headers in [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] serializeValueWithHeaders(final byte[] value, final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ final HeadersSerializer.PreSerializedHeaders preSerializedHeaders = HeadersSerializer.prepareSerialization(headers);
+ final byte[] rawHeaders = HeadersSerializer.serialize(
+ preSerializedHeaders,
+ ByteBuffer.allocate(preSerializedHeaders.requiredBufferSizeForHeaders)
+ ).array();
+
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+ ByteUtils.writeVarint(rawHeaders.length, out);
+ out.write(rawHeaders);
+ out.write(value);
+ return baos.toByteArray();
+ } catch (final IOException e) {
+ throw new RuntimeException("Failed to serialize value with headers", e);
+ }
+ }
+
+ /**
+ * Helper method to extract raw value from [headers][value] format.
+ * This is used for window stores with headers where the format is:
+ * [headersSize(varint)][headersBytes][value]
+ */
+ private static byte[] extractRawValue(final byte[] valueWithHeaders) {
Review Comment:
Should this go into the new `Utils` class we added? Or maybe it already exists there?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21780:
URL: https://github.com/apache/kafka/pull/21780#issuecomment-4086327823
Merged to meet feature freeze deadline -- pending comments are minor, and we can do a small follow up PR to address it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #21780:
URL: https://github.com/apache/kafka/pull/21780
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2944676741
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBMigratingWindowStoreWithHeaders.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-value store with headers support based on RocksDB for window stores.
+ * <p>
+ * This store provides a migration path from plain {@link RocksDBStore} (DEFAULT column family)
+ * to a headers-aware column family ({@code windowKeyValueWithHeaders}). It uses
+ * {@link DualColumnFamilyAccessor} for lazy migration when legacy data exists in the DEFAULT CF.
+ * <p>
+ * Value format:
+ * <ul>
+ * <li>Old format (DEFAULT CF): {@code [value]}</li>
+ * <li>New format (HEADERS CF): {@code [headersSize(varint)][headersBytes][value]}</li>
+ * </ul>
+ * <p>
+ * This is similar to {@link RocksDBMigratingSessionStoreWithHeaders} but for window stores.
+ * Unlike timestamped stores, window stores don't include timestamp in the value (it's in the key).
+ */
+public class RocksDBMigratingWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
Review Comment:
```suggestion
public class RocksDBWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
```
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/AbstractRocksDBTimeOrderedSegmentedBytesStore.java:
##########
@@ -113,14 +113,23 @@ public KeyValue<Bytes, byte[]> next() {
protected abstract Bytes getBaseKey(final Bytes indexKey);
}
+ @SuppressWarnings("unchecked")
AbstractRocksDBTimeOrderedSegmentedBytesStore(final String name,
final String metricsScope,
final long retention,
final long segmentInterval,
final KeySchema baseKeySchema,
final Optional<KeySchema> indexKeySchema) {
- super(name, baseKeySchema, indexKeySchema,
- new KeyValueSegments(name, metricsScope, retention, segmentInterval), retention);
+ this(name, retention, baseKeySchema, indexKeySchema,
+ (AbstractSegments<S>) new KeyValueSegments(name, metricsScope, retention, segmentInterval));
+ }
+
+ AbstractRocksDBTimeOrderedSegmentedBytesStore(final String name,
Review Comment:
Not sure why we add a new constructor? Seems we should rather just update the existing one?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBMigratingWindowStoreWithHeaders.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-value store with headers support based on RocksDB for window stores.
+ * <p>
+ * This store provides a migration path from plain {@link RocksDBStore} (DEFAULT column family)
+ * to a headers-aware column family ({@code windowKeyValueWithHeaders}). It uses
+ * {@link DualColumnFamilyAccessor} for lazy migration when legacy data exists in the DEFAULT CF.
+ * <p>
+ * Value format:
+ * <ul>
+ * <li>Old format (DEFAULT CF): {@code [value]}</li>
+ * <li>New format (HEADERS CF): {@code [headersSize(varint)][headersBytes][value]}</li>
+ * </ul>
+ * <p>
+ * This is similar to {@link RocksDBMigratingSessionStoreWithHeaders} but for window stores.
+ * Unlike timestamped stores, window stores don't include timestamp in the value (it's in the key).
+ */
+public class RocksDBMigratingWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+ private static final Logger log = LoggerFactory.getLogger(RocksDBMigratingWindowStoreWithHeaders.class);
+
+ static final byte[] WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME =
+ "windowKeyValueWithHeaders".getBytes(StandardCharsets.UTF_8);
+
+ RocksDBMigratingWindowStoreWithHeaders(final String name,
+ final String parentDir,
+ final RocksDBMetricsRecorder metricsRecorder) {
+ super(name, parentDir, metricsRecorder);
+ }
+
+ @Override
+ void openRocksDB(final DBOptions dbOptions,
+ final ColumnFamilyOptions columnFamilyOptions) {
+ final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+ dbOptions,
+ new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+ new ColumnFamilyDescriptor(WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME, columnFamilyOptions),
+ new ColumnFamilyDescriptor(OFFSETS_COLUMN_FAMILY_NAME, columnFamilyOptions)
+ );
+ final ColumnFamilyHandle noHeadersColumnFamily = columnFamilies.get(0);
+ final ColumnFamilyHandle withHeadersColumnFamily = columnFamilies.get(1);
+ final ColumnFamilyHandle offsetsCf = columnFamilies.get(2);
+
+ // Check if DEFAULT CF has data (upgrade from old format without headers)
+ try (final RocksIterator noHeadersIter = db.newIterator(noHeadersColumnFamily)) {
+ noHeadersIter.seekToFirst();
+ if (noHeadersIter.isValid()) {
+ log.info("Opening window store {} in upgrade mode from plain value format", name);
+ // Migrate from [value] to [headers][value]
+ // Add empty headers prefix [0x00] to plain value
+ cfAccessor = new DualColumnFamilyAccessor(
+ offsetsCf,
+ noHeadersColumnFamily,
+ withHeadersColumnFamily,
+ RocksDBMigratingWindowStoreWithHeaders::addEmptyHeadersPrefix,
+ this,
+ open
+ );
+ } else {
+ log.info("Opening window store {} in regular headers-aware mode", name);
+ cfAccessor = new SingleColumnFamilyAccessor(offsetsCf, withHeadersColumnFamily);
+ noHeadersColumnFamily.close();
+ }
+ }
+ }
+
+ /**
+ * Converts plain value to value with empty headers prefix.
+ * <p>
+ * For window stores, the legacy format is: [value]
+ * The new format is: [headersSize(varint)][headersBytes][value]
+ * <p>
+ * This method adds empty headers prefix to the plain value.
+ * Empty headers: headersSize = 0 (encoded as [0x00])
+ *
+ * @param plainValue the plain value bytes
+ * @return the value with empty headers prefix: [0x00][plainValue]
+ */
+ private static byte[] addEmptyHeadersPrefix(final byte[] plainValue) {
Review Comment:
Do we need to add it? Seems it's the same as `HeaderBytesStore.convertToHeaderFormat` ?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowSegmentedBytesStoreWithHeaders.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.processor.internals.ChangelogRecordDeserializationHelper;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.internals.PrefixedWindowKeySchemas.KeyFirstWindowKeySchema;
+import org.apache.kafka.streams.state.internals.PrefixedWindowKeySchemas.TimeFirstWindowKeySchema;
+
+import org.rocksdb.RocksDBException;
+import org.rocksdb.WriteBatch;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * A RocksDB-backed time-ordered segmented bytes store with headers support for window stores.
+ * <p>
+ * This store extends {@link AbstractRocksDBTimeOrderedSegmentedBytesStore} and uses
+ * {@link WindowSegmentsWithHeaders} to manage segments with full header support,
+ * including Column Family management and lazy migration from legacy formats.
+ * <p>
+ * The store maintains a dual-schema architecture:
+ * <ul>
+ * <li>Base store: Time-first schema for efficient time-range queries (fetchAll)</li>
+ * <li>Index store (optional): Key-first schema for efficient key-based queries (fetch)</li>
+ * </ul>
+ * <p>
+ * Headers are managed at the segment level by {@link WindowSegmentWithHeaders}.
+ * <p>
+ * Value format (timestamp is in the key, not in the value):
+ * <ul>
+ * <li>Old format: {@code [value]}</li>
+ * <li>New format: {@code [headers][value]}</li>
+ * </ul>
+ *
+ * @see RocksDBTimeOrderedWindowStore
+ * @see WindowSegmentsWithHeaders
+ * @see WindowSegmentWithHeaders
+ */
+class RocksDBTimeOrderedWindowSegmentedBytesStoreWithHeaders
+ extends AbstractRocksDBTimeOrderedSegmentedBytesStore<WindowSegmentWithHeaders> {
+
+ private class WindowKeySchemaIndexToBaseStoreIterator extends IndexToBaseStoreIterator {
Review Comment:
Seems this duplicate `RocksDBTimeOrderedWindowSegmentedBytesStore.WindowKeySchemaIndexToBaseStoreIterator` ? I would re-use the existing one (if we think it should not be a nested class for reuse, let's extract it)
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowSegmentedBytesStoreWithHeaders.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.clients.consumer.ConsumerRecord;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.errors.ProcessorStateException;
+import org.apache.kafka.streams.processor.internals.ChangelogRecordDeserializationHelper;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.internals.PrefixedWindowKeySchemas.KeyFirstWindowKeySchema;
+import org.apache.kafka.streams.state.internals.PrefixedWindowKeySchemas.TimeFirstWindowKeySchema;
+
+import org.rocksdb.RocksDBException;
+import org.rocksdb.WriteBatch;
+
+import java.util.Collection;
+import java.util.HashMap;
+import java.util.Map;
+import java.util.Optional;
+
+/**
+ * A RocksDB-backed time-ordered segmented bytes store with headers support for window stores.
+ * <p>
+ * This store extends {@link AbstractRocksDBTimeOrderedSegmentedBytesStore} and uses
+ * {@link WindowSegmentsWithHeaders} to manage segments with full header support,
+ * including Column Family management and lazy migration from legacy formats.
+ * <p>
+ * The store maintains a dual-schema architecture:
+ * <ul>
+ * <li>Base store: Time-first schema for efficient time-range queries (fetchAll)</li>
+ * <li>Index store (optional): Key-first schema for efficient key-based queries (fetch)</li>
+ * </ul>
+ * <p>
+ * Headers are managed at the segment level by {@link WindowSegmentWithHeaders}.
+ * <p>
+ * Value format (timestamp is in the key, not in the value):
+ * <ul>
+ * <li>Old format: {@code [value]}</li>
+ * <li>New format: {@code [headers][value]}</li>
+ * </ul>
+ *
+ * @see RocksDBTimeOrderedWindowStore
+ * @see WindowSegmentsWithHeaders
+ * @see WindowSegmentWithHeaders
+ */
+class RocksDBTimeOrderedWindowSegmentedBytesStoreWithHeaders
Review Comment:
Not sure if we need to introduce this class? It seems to be the same as `RocksDBTimeOrderedWindowSegmentedBytesStore`
If we change `RocksDBTimeOrderedWindowSegmentedBytesStore` from hard-coded `KeyValueSegment` to `<S extends Segment>` it seems we can re-use it for both case, ie, `KeyValueSegment` and the newly added `WindowSegmentWithHeaders` ?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBMigratingWindowStoreWithHeaders.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-value store with headers support based on RocksDB for window stores.
+ * <p>
+ * This store provides a migration path from plain {@link RocksDBStore} (DEFAULT column family)
+ * to a headers-aware column family ({@code windowKeyValueWithHeaders}). It uses
+ * {@link DualColumnFamilyAccessor} for lazy migration when legacy data exists in the DEFAULT CF.
+ * <p>
+ * Value format:
+ * <ul>
+ * <li>Old format (DEFAULT CF): {@code [value]}</li>
+ * <li>New format (HEADERS CF): {@code [headersSize(varint)][headersBytes][value]}</li>
+ * </ul>
+ * <p>
+ * This is similar to {@link RocksDBMigratingSessionStoreWithHeaders} but for window stores.
+ * Unlike timestamped stores, window stores don't include timestamp in the value (it's in the key).
+ */
+public class RocksDBMigratingWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
Review Comment:
We should also rename `RocksDBMigratingSessionStoreWithHeaders` to `RocksDBSessionStoreWithHeaders`
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBMigratingWindowStoreWithHeaders.java:
##########
@@ -0,0 +1,121 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.state.HeadersBytesStore;
+import org.apache.kafka.streams.state.internals.metrics.RocksDBMetricsRecorder;
+
+import org.rocksdb.ColumnFamilyDescriptor;
+import org.rocksdb.ColumnFamilyHandle;
+import org.rocksdb.ColumnFamilyOptions;
+import org.rocksdb.DBOptions;
+import org.rocksdb.RocksDB;
+import org.rocksdb.RocksIterator;
+import org.slf4j.Logger;
+import org.slf4j.LoggerFactory;
+
+import java.nio.ByteBuffer;
+import java.nio.charset.StandardCharsets;
+import java.util.List;
+
+/**
+ * A persistent key-value store with headers support based on RocksDB for window stores.
+ * <p>
+ * This store provides a migration path from plain {@link RocksDBStore} (DEFAULT column family)
+ * to a headers-aware column family ({@code windowKeyValueWithHeaders}). It uses
+ * {@link DualColumnFamilyAccessor} for lazy migration when legacy data exists in the DEFAULT CF.
+ * <p>
+ * Value format:
+ * <ul>
+ * <li>Old format (DEFAULT CF): {@code [value]}</li>
+ * <li>New format (HEADERS CF): {@code [headersSize(varint)][headersBytes][value]}</li>
+ * </ul>
+ * <p>
+ * This is similar to {@link RocksDBMigratingSessionStoreWithHeaders} but for window stores.
+ * Unlike timestamped stores, window stores don't include timestamp in the value (it's in the key).
+ */
+public class RocksDBMigratingWindowStoreWithHeaders extends RocksDBStore implements HeadersBytesStore {
+ private static final Logger log = LoggerFactory.getLogger(RocksDBMigratingWindowStoreWithHeaders.class);
+
+ static final byte[] WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME =
+ "windowKeyValueWithHeaders".getBytes(StandardCharsets.UTF_8);
+
+ RocksDBMigratingWindowStoreWithHeaders(final String name,
+ final String parentDir,
+ final RocksDBMetricsRecorder metricsRecorder) {
+ super(name, parentDir, metricsRecorder);
+ }
+
+ @Override
+ void openRocksDB(final DBOptions dbOptions,
+ final ColumnFamilyOptions columnFamilyOptions) {
+ final List<ColumnFamilyHandle> columnFamilies = openRocksDB(
+ dbOptions,
+ new ColumnFamilyDescriptor(RocksDB.DEFAULT_COLUMN_FAMILY, columnFamilyOptions),
+ new ColumnFamilyDescriptor(WINDOW_STORE_HEADERS_VALUES_COLUMN_FAMILY_NAME, columnFamilyOptions),
+ new ColumnFamilyDescriptor(OFFSETS_COLUMN_FAMILY_NAME, columnFamilyOptions)
+ );
+ final ColumnFamilyHandle noHeadersColumnFamily = columnFamilies.get(0);
+ final ColumnFamilyHandle withHeadersColumnFamily = columnFamilies.get(1);
+ final ColumnFamilyHandle offsetsCf = columnFamilies.get(2);
+
+ // Check if DEFAULT CF has data (upgrade from old format without headers)
+ try (final RocksIterator noHeadersIter = db.newIterator(noHeadersColumnFamily)) {
+ noHeadersIter.seekToFirst();
+ if (noHeadersIter.isValid()) {
+ log.info("Opening window store {} in upgrade mode from plain value format", name);
+ // Migrate from [value] to [headers][value]
+ // Add empty headers prefix [0x00] to plain value
+ cfAccessor = new DualColumnFamilyAccessor(
+ offsetsCf,
+ noHeadersColumnFamily,
+ withHeadersColumnFamily,
+ RocksDBMigratingWindowStoreWithHeaders::addEmptyHeadersPrefix,
+ this,
+ open
+ );
+ } else {
+ log.info("Opening window store {} in regular headers-aware mode", name);
+ cfAccessor = new SingleColumnFamilyAccessor(offsetsCf, withHeadersColumnFamily);
+ noHeadersColumnFamily.close();
+ }
+ }
+ }
+
+ /**
+ * Converts plain value to value with empty headers prefix.
+ * <p>
+ * For window stores, the legacy format is: [value]
+ * The new format is: [headersSize(varint)][headersBytes][value]
+ * <p>
+ * This method adds empty headers prefix to the plain value.
+ * Empty headers: headersSize = 0 (encoded as [0x00])
+ *
+ * @param plainValue the plain value bytes
+ * @return the value with empty headers prefix: [0x00][plainValue]
+ */
+ private static byte[] addEmptyHeadersPrefix(final byte[] plainValue) {
+ if (plainValue == null) {
+ return null;
+ }
+ // For empty headers: headersSize = varint(0) = [0x00]
+ return ByteBuffer.allocate(1 + plainValue.length)
Review Comment:
I think we should reuse the existing code. In doubt we should not add this helper here anyway, but to `HeadersBytesStore` as central place where all these converts should live.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20317: Enable time-ordered header window store in DSL [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21780:
URL: https://github.com/apache/kafka/pull/21780#discussion_r2955352432
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/RocksDBTimeOrderedWindowStore.java:
##########
@@ -46,7 +46,7 @@ public class RocksDBTimeOrderedWindowStore
private StateStoreContext stateStoreContext;
RocksDBTimeOrderedWindowStore(
- final RocksDBTimeOrderedWindowSegmentedBytesStore store,
+ final AbstractRocksDBTimeOrderedSegmentedBytesStore<? extends Segment> store,
Review Comment:
Would allow us to use `AbstractRocksDBTimeOrderedSegmentedBytesStore<S>` here, too and ensuring that the types match up? `<?>` means "I don't care", but we do care that the segment type here, and the segment type of the super-class are the same.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org