You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "frankvicky (via GitHub)" <gi...@apache.org> on 2026/02/05 13:21:34 UTC
[PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
frankvicky opened a new pull request, #21408:
URL: https://github.com/apache/kafka/pull/21408
JIRA: [KAFKA-20121](https://issues.apache.org/jira/browse/KAFKA-20121)
This PR aims to add the `ValueTimestampHeaders` and its related serializer/deserializer as infrastructure of KIP-1271
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2781470451
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
>Seems we are using it only in tests?
No, we need it later in later PRs.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2789597205
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
> If the caller class needs to call it even only once, it needs to create an instance and then call it.
That's the question. Do we have such a case? -- My understanding would be, that a caller would need to use it regularly, and thus we would "trash" GC (even if not super expensive for very short lived object) using a static method, and it seems better if the caller uses a single long live instance of `HeaderDeserializer`?
We can of course also keep it for now and see how it goes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2773718207
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
Review Comment:
You forgot `public` here!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779572389
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+ }
+
+ @Test
+ public void shouldBeEqualWhenSameValues() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentValues() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make("different", TIMESTAMP, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentTimestamps() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP + 1, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentHeaders() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key2", "value2".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldBeEqualToItself() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(valueTimestampHeaders, valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHaveCorrectToString() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final String toString = valueTimestampHeaders.toString();
+ assertNotNull(toString);
+ assertTrue(toString.contains("value=" + VALUE));
+ assertTrue(toString.contains("timestamp=" + TIMESTAMP));
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+ assertEquals("key1", deserializedHeaders.lastHeader("key1").key());
+ assertArrayEquals("value1".getBytes(), deserializedHeaders.lastHeader("key1").value());
+ }
+
+ @Test
+ public void shouldDeserializeHeaderLazily() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertNull(valueTimestampHeaders.headers, "should be null before invocation of headers()");
+ valueTimestampHeaders.headers();
+ assertNotNull(valueTimestampHeaders.headers, "should be deserialize after invocation of headers()");
+ assertEquals(headers, valueTimestampHeaders.headers);
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(null, TIMESTAMP, rawHeaders);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHandleEmptyRawHeaders() {
+ final Headers headers = new RecordHeaders();
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(0, deserializedHeaders.toArray().length);
+ }
+
+ @Test
+ public void shouldBeEqualWhenCreatedWithMakeAndMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldBeEqualWhenBothCreatedWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldCallHeadersMultipleTimesWithoutError() {
Review Comment:
I was think of testing if there is any error during lazy initialization but it seems not very useful.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2775518509
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
Review Comment:
```suggestion
// skip headers plus timestamp
buffer.position(buffer.position() + headersSize + Long.BYTES);
```
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
Review Comment:
nit: add information about null/empty header and `byte[0]` case?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
Review Comment:
Should this be `private`?
```suggestion
public byte[] serialize(final String topic, final V plainValue, final long timestamp, final Headers headers) {
```
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+ }
+
+ @Test
+ public void shouldBeEqualWhenSameValues() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentValues() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make("different", TIMESTAMP, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentTimestamps() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP + 1, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentHeaders() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key2", "value2".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldBeEqualToItself() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(valueTimestampHeaders, valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHaveCorrectToString() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final String toString = valueTimestampHeaders.toString();
+ assertNotNull(toString);
+ assertTrue(toString.contains("value=" + VALUE));
+ assertTrue(toString.contains("timestamp=" + TIMESTAMP));
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+ assertEquals("key1", deserializedHeaders.lastHeader("key1").key());
+ assertArrayEquals("value1".getBytes(), deserializedHeaders.lastHeader("key1").value());
+ }
+
+ @Test
+ public void shouldDeserializeHeaderLazily() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertNull(valueTimestampHeaders.headers, "should be null before invocation of headers()");
+ valueTimestampHeaders.headers();
+ assertNotNull(valueTimestampHeaders.headers, "should be deserialize after invocation of headers()");
+ assertEquals(headers, valueTimestampHeaders.headers);
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(null, TIMESTAMP, rawHeaders);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHandleEmptyRawHeaders() {
+ final Headers headers = new RecordHeaders();
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(0, deserializedHeaders.toArray().length);
+ }
+
+ @Test
+ public void shouldBeEqualWhenCreatedWithMakeAndMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldBeEqualWhenBothCreatedWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldCallHeadersMultipleTimesWithoutError() {
Review Comment:
How could there be an error? Is this test useful?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+
+ return readBytes(buffer, buffer.remaining());
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
+ }
+
+ /**
+ * Extract headers from serialized ValueTimestampHeaders.
+ */
+ static Headers headers(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+
+ return HeadersDeserializer.deserialize(rawHeaders);
Review Comment:
If we need/want to deserialize here, might be better to add a `final static HeadersDeserializer HEADERS_DESERIALIZE = new HeadersDeserializer()` object to the class to avoid the static helper methods (reducing boiler plate code)
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+ assertEquals(TIMESTAMP, extractedTimestamp);
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+ assertNotNull(extractedHeaders);
+ assertEquals(2, extractedHeaders.toArray().length);
+ assertArrayEquals("value1".getBytes(), extractedHeaders.lastHeader("key1").value());
+ assertArrayEquals("value2".getBytes(), extractedHeaders.lastHeader("key2").value());
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, null);
+ assertNull(deserialized);
+ }
+
+ @Test
+ public void shouldReturnNullForRawValueOfNull() {
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(null);
Review Comment:
same question
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ Header[] headersArray = deserializedHeaders.toArray();
+ assertEquals(3, headersArray.length);
+
+ for (int i = 0; i < headersArray.length; i++) {
+ Header header = headersArray[i];
+ assertEquals("key" + i, header.key());
+ assertArrayEquals(("value" + i).getBytes(), header.value());
+ }
+ }
+
+ @Test
+ public void shouldReturnNullWhenSerializingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.makeAllowNullable(null, 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNull(serialized, "Serializer should return null when value is null");
+
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+ assertNull(deserialized, "Deserializer should return null when input is null");
+ }
+
+ @Test
+ public void shouldDeserializeWithHeaderContainingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", null);
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Header header = deserialized.headers().lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertNull(header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithLargeValue() {
+ final StringBuilder largeValue = new StringBuilder();
+ largeValue.append("x".repeat(10000));
+
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(largeValue.toString(), 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(largeValue.toString(), deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertEquals(1, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+
+ assertNotNull(rawValue);
+
+ try (Serde<String> stringSerde = Serdes.String()) {
+ final String value = stringSerde.deserializer().deserialize(TOPIC, rawValue);
+ assertEquals("test-value", value);
+ }
+ }
+
+ @Test
+ public void shouldReturnNullForRawValueWhenInputIsNull() {
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(null);
+ assertNull(rawValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final long timestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+
+ assertEquals(123456789L, timestamp);
+ }
+
+ @Test
+ public void shouldThrowExceptionWhenExtractingTimestampFromNull() {
+ // ByteBuffer.wrap() throws NullPointerException for null input
+ assertThrows(NullPointerException.class, () ->
+ ValueTimestampHeadersDeserializer.timestamp(null)
+ );
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+
+ assertNotNull(extractedHeaders);
+ assertEquals(2, extractedHeaders.toArray().length);
+ assertArrayEquals("value1".getBytes(), extractedHeaders.lastHeader("key1").value());
+ assertArrayEquals("value2".getBytes(), extractedHeaders.lastHeader("key2").value());
+ }
+
+ @Test
+ public void shouldExtractEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+
+ assertNotNull(extractedHeaders);
+ assertEquals(0, extractedHeaders.toArray().length);
+ }
+
+ @Test
+ public void shouldReturnNullWhenExtractingHeadersFromNull() {
+ final Headers headers = ValueTimestampHeadersDeserializer.headers(null);
+ assertNull(headers);
+ }
+
+ @Test
+ public void shouldThrowExceptionWhenDataIsTooShort() {
+ // Create malformed data: only headersSize varint, no actual headers or timestamp
+ final byte[] malformedData = new byte[] {0x02}; // headersSize = 1 but no data follows
+
+ assertThrows(IllegalArgumentException.class, () ->
+ deserializer.deserialize(TOPIC, malformedData),
+ "Should throw IllegalArgumentException for malformed data"
+ );
+ }
+
+ @Test
+ public void shouldThrowExceptionWhenHeadersSizeIsInconsistent() {
+ // Create data with headersSize = 10 but not enough actual data
+ final byte[] malformedData = new byte[] {
+ 0x14, // headersSize = 10 (ZigZag encoding)
+ 0x00, 0x00 // Only 2 bytes when 10 + 8 (timestamp) are expected
+ };
+
+ assertThrows(IllegalArgumentException.class, () ->
Review Comment:
Should this be a `SerializationException` ?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
Review Comment:
Super nit: we pass in argument in order value, ts, headers -- why do we serialize ts after headers? (This break my reading flow of the code...)
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+ assertEquals(TIMESTAMP, extractedTimestamp);
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+ assertNotNull(extractedHeaders);
+ assertEquals(2, extractedHeaders.toArray().length);
+ assertArrayEquals("value1".getBytes(), extractedHeaders.lastHeader("key1").value());
+ assertArrayEquals("value2".getBytes(), extractedHeaders.lastHeader("key2").value());
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
Review Comment:
This test already existing in `ValueTimestampHeadersDeserializerTest`
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
Review Comment:
nit: should we update comment about `byte[0]` case?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
+
+ // Format: [headersSize(varint)][headersBytes][timestamp(8)][value]
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream out = new DataOutputStream(baos)) {
+
+ ByteUtils.writeVarint(rawHeaders.length, out); // headersSize
+ out.write(rawHeaders); // [count][header1][header2]...
Review Comment:
nit: `byte[0]` case?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
Not sure why we need this? Seems we are using it only in tests? Why can't we just create an instance in the test code?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
+
+ // Format: [headersSize(varint)][headersBytes][timestamp(8)][value]
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream out = new DataOutputStream(baos)) {
+
+ ByteUtils.writeVarint(rawHeaders.length, out); // headersSize
Review Comment:
```suggestion
ByteUtils.writeVarint(rawHeaders.length, out); // headersSize (may be 0)
```
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+ assertEquals(TIMESTAMP, extractedTimestamp);
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
Review Comment:
Same question
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
If KS is using this method only internally, wondering if we need to expose it publicly?
Not sure from top of my head what options we have? Could we make this class non-final, and add a non-public sub-class maybe, to add this functionality?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
Review Comment:
```suggestion
.add("key1", "value2".getBytes())
```
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ Header[] headersArray = deserializedHeaders.toArray();
+ assertEquals(3, headersArray.length);
+
+ for (int i = 0; i < headersArray.length; i++) {
+ Header header = headersArray[i];
+ assertEquals("key" + i, header.key());
+ assertArrayEquals(("value" + i).getBytes(), header.value());
+ }
+ }
+
+ @Test
+ public void shouldReturnNullWhenSerializingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.makeAllowNullable(null, 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNull(serialized, "Serializer should return null when value is null");
+
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+ assertNull(deserialized, "Deserializer should return null when input is null");
+ }
+
+ @Test
+ public void shouldDeserializeWithHeaderContainingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", null);
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Header header = deserialized.headers().lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertNull(header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithLargeValue() {
Review Comment:
Not sure what the purpose of this test is?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
Review Comment:
Should we also test this for `makeAllowNullable(null, TIMESTAMP, headers)` case?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
Review Comment:
Should be ok? Seems we are getting empty byte array (ie, `new byte[0]`) back, which the `HeadersDeserializer` can handle correctly.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ Header[] headersArray = deserializedHeaders.toArray();
+ assertEquals(3, headersArray.length);
+
+ for (int i = 0; i < headersArray.length; i++) {
+ Header header = headersArray[i];
+ assertEquals("key" + i, header.key());
+ assertArrayEquals(("value" + i).getBytes(), header.value());
+ }
+ }
+
+ @Test
+ public void shouldReturnNullWhenSerializingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.makeAllowNullable(null, 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNull(serialized, "Serializer should return null when value is null");
+
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+ assertNull(deserialized, "Deserializer should return null when input is null");
+ }
+
+ @Test
+ public void shouldDeserializeWithHeaderContainingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", null);
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Header header = deserialized.headers().lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertNull(header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithLargeValue() {
+ final StringBuilder largeValue = new StringBuilder();
+ largeValue.append("x".repeat(10000));
+
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(largeValue.toString(), 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(largeValue.toString(), deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertEquals(1, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+
+ assertNotNull(rawValue);
+
+ try (Serde<String> stringSerde = Serdes.String()) {
+ final String value = stringSerde.deserializer().deserialize(TOPIC, rawValue);
+ assertEquals("test-value", value);
+ }
+ }
+
+ @Test
+ public void shouldReturnNullForRawValueWhenInputIsNull() {
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(null);
+ assertNull(rawValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final long timestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+
+ assertEquals(123456789L, timestamp);
+ }
+
+ @Test
+ public void shouldThrowExceptionWhenExtractingTimestampFromNull() {
+ // ByteBuffer.wrap() throws NullPointerException for null input
+ assertThrows(NullPointerException.class, () ->
+ ValueTimestampHeadersDeserializer.timestamp(null)
+ );
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+
+ assertNotNull(extractedHeaders);
+ assertEquals(2, extractedHeaders.toArray().length);
+ assertArrayEquals("value1".getBytes(), extractedHeaders.lastHeader("key1").value());
+ assertArrayEquals("value2".getBytes(), extractedHeaders.lastHeader("key2").value());
+ }
+
+ @Test
+ public void shouldExtractEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
+
+ assertNotNull(extractedHeaders);
+ assertEquals(0, extractedHeaders.toArray().length);
+ }
+
+ @Test
+ public void shouldReturnNullWhenExtractingHeadersFromNull() {
+ final Headers headers = ValueTimestampHeadersDeserializer.headers(null);
+ assertNull(headers);
+ }
+
+ @Test
+ public void shouldThrowExceptionWhenDataIsTooShort() {
+ // Create malformed data: only headersSize varint, no actual headers or timestamp
+ final byte[] malformedData = new byte[] {0x02}; // headersSize = 1 but no data follows
+
+ assertThrows(IllegalArgumentException.class, () ->
Review Comment:
Should this be a `SerializationException`?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
Review Comment:
```suggestion
public byte[] serialize(final String topic, final ValueTimestampHeaders<V> valueTimestampHeaders) {
```
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
Review Comment:
not: add information about null/empty header and `byte[0]` case?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+
+ return readBytes(buffer, buffer.remaining());
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
+ }
+
+ /**
+ * Extract headers from serialized ValueTimestampHeaders.
+ */
+ static Headers headers(final byte[] rawValueTimestampHeaders) {
Review Comment:
Seem we add static helpers for `rawValue` (ie, not deserialized), and `timestamp` (deserialized) and `headers` (again deserialized).
Timestamp is kinda special case -- but wondering why we have to mix of "raw value" and "headers" -- instead of have both "raw" or both deserialized?
As these are all internal helper method, for the value cause we could just pass the required value deserializer as parameter?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+ }
+
+ @Test
+ public void shouldBeEqualWhenSameValues() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentValues() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make("different", TIMESTAMP, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentTimestamps() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP + 1, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentHeaders() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key2", "value2".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldBeEqualToItself() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(valueTimestampHeaders, valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHaveCorrectToString() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final String toString = valueTimestampHeaders.toString();
+ assertNotNull(toString);
+ assertTrue(toString.contains("value=" + VALUE));
+ assertTrue(toString.contains("timestamp=" + TIMESTAMP));
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+ assertEquals("key1", deserializedHeaders.lastHeader("key1").key());
+ assertArrayEquals("value1".getBytes(), deserializedHeaders.lastHeader("key1").value());
+ }
+
+ @Test
+ public void shouldDeserializeHeaderLazily() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertNull(valueTimestampHeaders.headers, "should be null before invocation of headers()");
+ valueTimestampHeaders.headers();
+ assertNotNull(valueTimestampHeaders.headers, "should be deserialize after invocation of headers()");
+ assertEquals(headers, valueTimestampHeaders.headers);
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(null, TIMESTAMP, rawHeaders);
Review Comment:
Do we need a `makeAllowNullableWithRawHeaders`, too? (Would be good to know to avoid updating the KIP too many times)
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+
+ return readBytes(buffer, buffer.remaining());
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
Review Comment:
`null` check is missing -- guess we would return `-1L` for `null`?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+ }
+
+ @Test
+ public void shouldBeEqualWhenSameValues() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentValues() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make("different", TIMESTAMP, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentTimestamps() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP + 1, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentHeaders() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key2", "value2".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldBeEqualToItself() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(valueTimestampHeaders, valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHaveCorrectToString() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final String toString = valueTimestampHeaders.toString();
+ assertNotNull(toString);
+ assertTrue(toString.contains("value=" + VALUE));
+ assertTrue(toString.contains("timestamp=" + TIMESTAMP));
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+ assertEquals("key1", deserializedHeaders.lastHeader("key1").key());
+ assertArrayEquals("value1".getBytes(), deserializedHeaders.lastHeader("key1").value());
+ }
+
+ @Test
+ public void shouldDeserializeHeaderLazily() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertNull(valueTimestampHeaders.headers, "should be null before invocation of headers()");
+ valueTimestampHeaders.headers();
+ assertNotNull(valueTimestampHeaders.headers, "should be deserialize after invocation of headers()");
+ assertEquals(headers, valueTimestampHeaders.headers);
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(null, TIMESTAMP, rawHeaders);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHandleEmptyRawHeaders() {
Review Comment:
Not sure if we need this test? Seems to not give any signal? This test seems to repeat what we already test (ie, re-tests the internally use `HeadersDeserializer`) -- there is no logic in `makeWithRawHeaders` that would treat "empty" differently.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
Review Comment:
```suggestion
.add("key0", "value1".getBytes())
```
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
Review Comment:
Why does this need to be `volatile` ?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
Review Comment:
Add duplicate keys again?
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ if (headers == null) {
+ synchronized (this) {
Review Comment:
Why do we need to make this thread safe?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
Review Comment:
High level comment. This an the blow helpers are not use yet in this PR -- do we need them later? Ok to add now, if we need them; just double checking because it's unclear from the PR.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
Review Comment:
This test is for `ValueTimestampHeadersDeserializer.timestamp` as it seems? Should it go into `ValueTimestampHeadersDeserializerTest`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2776428431
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
Review Comment:
Yes, it should be fine.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2774747303
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
Review Comment:
if `headersSize` is 0, what happens here?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2776211408
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
Review Comment:
High level comment. This and the blow helpers are not use yet in this PR -- do we need them later? Ok to add now, if we need them; just double checking because it's unclear from the PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2786277415
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
>can't the caller code not just create an instance of HeaderDeserializer
what is the reason that you prefer non-static? I'm OK, just as learning purpose I ask. If the caller class needs to call it even only once, it needs to create an instance and then call it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2776433244
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+
+ return readBytes(buffer, buffer.remaining());
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
Review Comment:
@aliehsaeedii has same comment, here is my thought. WDYT?
https://github.com/apache/kafka/pull/21408#discussion_r2770938049
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2787110002
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key1", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
Review Comment:
Make sense.
I'm thinking we could replace this test with testing serializing null value:
```
@Test
public void shouldReturnNullWhenSerializingNullValue() {
final ValueTimestampHeaders<String> valueTimestampHeaders =
ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, new RecordHeaders());
final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
assertNull(serialized);
}
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2785175107
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key1", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
Review Comment:
Given that `make()` replaces `null` with `RecordHeader` (and we should verify this in `ValueTimestampHeaderTest`), this test seem not to give any signal, as `null` would never reach the serializer?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779555089
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+
+ return readBytes(buffer, buffer.remaining());
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
+ }
+
+ /**
+ * Extract headers from serialized ValueTimestampHeaders.
+ */
+ static Headers headers(final byte[] rawValueTimestampHeaders) {
Review Comment:
Good catch, I think we could have these helper methods return deserialized value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770851535
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -113,30 +127,21 @@ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
}
final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
- final int headerSize = ByteUtils.readVarint(buffer);
- buffer.position(buffer.position() + headerSize + Long.BYTES);
-
- final byte[] rawValue = new byte[buffer.remaining()];
- buffer.get(rawValue);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
- return rawValue;
+ return readBytes(buffer, buffer.remaining());
}
/**
* Extract timestamp from serialized ValueTimestampHeaders.
*/
static long timestamp(final byte[] rawValueTimestampHeaders) {
- if (rawValueTimestampHeaders == null) {
Review Comment:
Why did you remove this `if (rawValueTimestampHeaders == null)` check?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -104,6 +98,26 @@ public void setIfUnset(final SerdeGetter getter) {
initNullableDeserializer(valueDeserializer, getter);
}
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
Review Comment:
Not sure if `SerializationExcpetion` is more specific and fits more
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2778078789
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
Was thinking about this more. And have a better idea (I believe).
It's actually a good thing to have the public facing `ValueTimestampHeaders` being a `final` class so might be better to not change. Overall, lazy deserialization should be an impl detail the user does not see, and maybe even `ValueTimestampHeaders` does not need to know about it -- it seems to be a leaky abstraction if we add a `makeWithRawHeaders()` method (even if we add it on a different, non-public class).
It seems better to fully embed lazy deserialization inside `HeadersDeserializer`. I would propose to change its `deserialize(...)` method:
```
public Headers deserialize(final String topic, final byte[] data) {
return new LazyRecordHeaders(data);
}
```
We just return a new "headers" object, and push the deserialization inside `LazyRecordHeaders` (a new class we add: not sure if this extend `RecordHeaders` or just implement `Headers` interface directly) itself. The constructor would just store `data` byte array, and when the object is accessed, we deserialize internally.
Thoughts? \cc @aliehsaeedii
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779504104
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
Review Comment:
I suppose we should keep it, both window/KV state store will need these methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on PR #21408:
URL: https://github.com/apache/kafka/pull/21408#issuecomment-3853750710
This should only be merged after #21401
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770220501
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersSerializer.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Serializer for Kafka Headers.
+ * <p>
+ * Serialization format (per KIP-1271):
+ * [NumHeaders(varint)][Header1][Header2]...
+ * <p>
+ * Each header:
+ * [KeyLength(varint)][KeyBytes(UTF-8)][ValueLength(varint)][ValueBytes]
+ * <p>
+ * Note: ValueLength is -1 for null values (encoded as varint).
+ * All integers are encoded as varints (signed varint encoding).
+ * <p>
+ * This serializer produces the headersBytes portion. The headersSize prefix
+ * is added by the outer serializer (e.g., ValueTimestampHeadersSerializer).
+ * <p>
+ * This is used by KIP-1271 to serialize headers for storage in state stores.
+ */
+public class HeadersSerializer implements Serializer<Headers> {
+
+ /**
+ * Serializes headers into a byte array using varint encoding per KIP-1271.
+ * <p>
+ * The output format is [count][header1][header2]... without a size prefix.
+ * The size prefix is added by the outer serializer that uses this.
+ *
+ * @param topic topic associated with data
+ * @param headers the headers to serialize (can be null)
+ * @return the serialized byte array
+ */
+ @Override
+ public byte[] serialize(final String topic, final Headers headers) {
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+
+ final Header[] headerArray = (headers == null) ? new Header[0] : headers.toArray();
+ ByteUtils.writeVarint(headerArray.length, out);
+
+ for (final Header header : headerArray) {
+ final byte[] keyBytes = header.key().getBytes(StandardCharsets.UTF_8);
Review Comment:
Yes.
The references:
https://github.com/apache/kafka/blob/9d5bbf58277133ea2793cdbfbb75e68beac16117/clients/src/main/java/org/apache/kafka/common/header/Header.java#L24-L29
https://github.com/apache/kafka/blob/9d5bbf58277133ea2793cdbfbb75e68beac16117/clients/src/main/java/org/apache/kafka/common/header/internals/RecordHeader.java#L32-L41
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2778078789
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
Was thinking about this more. And have a better idea (I believe).
It's actually a good thing to have the public facing `ValueTimestampHeaders` being a `final` class so might be better to not change. Overall, lazy deserialization should be an impl detail the user does not see, and maybe even `ValueTimestampHeaders` does not need to know about it -- it seems to be a leaky abstraction if we add a `makeWithRawHeaders()` method (even if we add it on a different, non-public class).
It seems better to fully embed lazy deserialization inside `HeadersDeserializer`. I would propose to change its `deserialize(...)` method:
```
public Headers deserialize(final String topic, final byte[] data) {
return new LazyRecordHeaders(data);
}
```
We just return a new "headers" object, and push the deserialization inside `LazyRecordHeaders` (a new class we add: not sure if this extend `RecordHeaders` or just implement `Headers` interface directly) itself. The constructor would just store `data` byte array (similar to what we do right now with `makeWithRawHeaders` , and we only really deserialize when the `LazyRecordHeaders` object is accessed.
Thoughts? \cc @aliehsaeedii
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2785076023
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
Wondering why we would need it -- can't the caller code not just create an instance of `HeaderDeserializer`? Seems to be a little bit odd, to have `static` method would would create in instance of "itself" which is short lived.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770205073
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ if (headers == null && rawHeaders != null) {
+ headers = HeadersDeserializer.deserialize(rawHeaders);
+ }
+ return headers != null ? headers : new RecordHeaders();
Review Comment:
Nice catch
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770282972
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
Review Comment:
Nope, for `rawValue`, I follow the pattern from `ValueAndTimestampDeserializer`.
I will unify them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2781705968
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
@mjsax @frankvicky
This is really an architectural improvement. Right now, `ValueTimestampHeaders` has both `headers` and `rawHeaders `fields and `makeWithRawHeaders()` exposes lazy deserialization as part of the public contract which is not a clean design as you mentioned. I got the issue and agree with you about that.
Regarddding the solution: I can not think of anything better. The trade-off is adding a new class, but that's worth it for the cleaner separation of concerns.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2781782148
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,263 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersSerializer;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+ }
+
+ @Test
+ public void shouldBeEqualWhenSameValues() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ assertEquals(valueTimestampHeaders1.hashCode(), valueTimestampHeaders2.hashCode());
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentValues() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make("different", TIMESTAMP, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentTimestamps() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP + 1, headers);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldNotBeEqualWhenDifferentHeaders() {
+ final Headers headers1 = new RecordHeaders().add("key1", "value1".getBytes());
+ final Headers headers2 = new RecordHeaders().add("key2", "value2".getBytes());
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders1 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers1);
+ final ValueTimestampHeaders<String> valueTimestampHeaders2 = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers2);
+
+ assertNotEquals(valueTimestampHeaders1, valueTimestampHeaders2);
+ }
+
+ @Test
+ public void shouldBeEqualToItself() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertEquals(valueTimestampHeaders, valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldHaveCorrectToString() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final String toString = valueTimestampHeaders.toString();
+ assertNotNull(toString);
+ assertTrue(toString.contains("value=" + VALUE));
+ assertTrue(toString.contains("timestamp=" + TIMESTAMP));
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+
+ final Headers deserializedHeaders = valueTimestampHeaders.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+ assertEquals("key1", deserializedHeaders.lastHeader("key1").key());
+ assertArrayEquals("value1".getBytes(), deserializedHeaders.lastHeader("key1").value());
+ }
+
+ @Test
+ public void shouldDeserializeHeaderLazily() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(VALUE, TIMESTAMP, rawHeaders);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertNull(valueTimestampHeaders.headers, "should be null before invocation of headers()");
+ valueTimestampHeaders.headers();
+ assertNotNull(valueTimestampHeaders.headers, "should be deserialize after invocation of headers()");
+ assertEquals(headers, valueTimestampHeaders.headers);
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMakeWithRawHeaders() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final byte[] rawHeaders = new HeadersSerializer().serialize("", headers);
+
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.makeWithRawHeaders(null, TIMESTAMP, rawHeaders);
Review Comment:
We should postpone this till we start dealing with lazy deserialization issue.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #21408:
URL: https://github.com/apache/kafka/pull/21408
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2786927780
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
Review Comment:
Totally forget we have it 🤣
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2778078789
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
Was thinking about this more. And have a better idea (I believe).
It's actually a good thing to have the public facing `ValueTimestampHeaders` being a `final` class, and overall, lazy deserialization should be an impl detail the user does not see, and maybe even `ValueTimestampHeaders` does not need to know about it -- it seems to be a leaky abstraction if we add a `makeWithRawHeaders()` method (even if we add it on a different, non-public class).
It seems better to fully embed lazy deserialization inside `HeadersDeserializer`. I would propose to change its `deserialize(...)` method:
```
public Headers deserialize(final String topic, final byte[] data) {
return new LazyRecordHeaders(data);
}
```
We just return a new "headers" object, and push the deserialization inside `LazyRecordHeaders` class itself. The constructor would just store `data` byte array, and when the object is accessed, we deserialize internally.
Thoughts? \cc @aliehsaeedii
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2785103084
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ return headers;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof ValueTimestampHeaders)) {
+ return false;
+ }
+ final ValueTimestampHeaders<?> that = (ValueTimestampHeaders<?>) o;
+ // Ensure headers are deserialized before comparison (lazy deserialization)
+ return timestamp == that.timestamp
+ && Objects.equals(value, that.value)
+ && Objects.equals(this.headers(), that.headers());
+ }
+
+ @Override
+ public int hashCode() {
+ // Ensure headers are deserialized before hashing (lazy deserialization)
+ return Objects.hash(value, timestamp, headers());
+ }
+
+ @Override
+ public String toString() {
+ return "ValueTimestampHeaders{" +
+ "value=" + value +
+ ", timestamp=" + timestamp +
+ ", headers=" + headers() +
Review Comment:
Same.
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ return headers;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof ValueTimestampHeaders)) {
+ return false;
+ }
+ final ValueTimestampHeaders<?> that = (ValueTimestampHeaders<?>) o;
+ // Ensure headers are deserialized before comparison (lazy deserialization)
+ return timestamp == that.timestamp
+ && Objects.equals(value, that.value)
+ && Objects.equals(this.headers(), that.headers());
+ }
+
+ @Override
+ public int hashCode() {
+ // Ensure headers are deserialized before hashing (lazy deserialization)
+ return Objects.hash(value, timestamp, headers());
Review Comment:
Same
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key1", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
Review Comment:
Given that `make() replaces `null` with `RecordHeader` (and we should verify this in `ValueTimestampHeaderTest`), this test seem not to give any signal, as `null` would never reach the serializer?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+
+ valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, null);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
Review Comment:
Seems to be the first test case so separation it with an empty line. We test `getValueOrNull` with object create with `make`, `makeAllowNullable`, and `null`-object.
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,278 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import java.util.Iterator;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertFalse;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
Review Comment:
```suggestion
public void shouldDeserializeNullToNull() {
```
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ return headers;
+ }
+
+ @Override
+ public boolean equals(final Object o) {
+ if (this == o) {
+ return true;
+ }
+ if (!(o instanceof ValueTimestampHeaders)) {
+ return false;
+ }
+ final ValueTimestampHeaders<?> that = (ValueTimestampHeaders<?>) o;
+ // Ensure headers are deserialized before comparison (lazy deserialization)
+ return timestamp == that.timestamp
+ && Objects.equals(value, that.value)
+ && Objects.equals(this.headers(), that.headers());
Review Comment:
If we remove the "lazy deserialization" from this class, we can use `this.headers` members variable directly -- no need to call the getter. (Also can remove the comment about it above.)
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+ return serialize(topic, valueTimestampHeaders.value(), valueTimestampHeaders.timestamp(), valueTimestampHeaders.headers());
+ }
+
+ public byte[] serialize(final String topic, final V plainValue, final long timestamp, final Headers headers) {
Review Comment:
nit: should this be `private`?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+ return serialize(topic, valueTimestampHeaders.value(), valueTimestampHeaders.timestamp(), valueTimestampHeaders.headers());
+ }
+
+ public byte[] serialize(final String topic, final V plainValue, final long timestamp, final Headers headers) {
+ if (plainValue == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, plainValue);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
+
+ // empty (byte[0]) for null/empty headers, or [count][header1][header2]... for non-empty
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers);
+
+ // Format: [headersSize(varint)][headersBytes][timestamp(8)][value]
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
Review Comment:
nit: add `final` (2x)
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,161 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
Review Comment:
Seem we can remove this constructor?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,128 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
Review Comment:
Nit: why `Serializer<Long>` instead of `LongSerializer`?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers in the format [count(varint)][header1][header2]... to be processed by HeadersDeserializer.
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+ private static final HeadersDeserializer HEADERS_DESERIALIZER = new HeadersDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
Review Comment:
nit: why not `LongDeserializer` ?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers in the format [count(varint)][header1][header2]... to be processed by HeadersDeserializer.
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+ private static final HeadersDeserializer HEADERS_DESERIALIZER = new HeadersDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final Headers headers = headersDeserializer.deserialize("", rawHeaders);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.make(value, timestamp, headers);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws SerializationException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new SerializationException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract value from serialized ValueTimestampHeaders.
+ */
+ static <V> V value(final byte[] rawValueTimestampHeaders, final Deserializer<V> deserializer) {
Review Comment:
nit: Should we really overload generic type `V` -- We use it already on the class, and the type here has nothing to do with the class type, as it's static method -- easy to mix up.
Should we use `T` instead to get a better distinction?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+
+ valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, null);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
Review Comment:
```suggestion
assertNull(ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
```
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,148 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key1", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
Review Comment:
This seems to test `ValueTimestampHeadersDeserializer.value` -- so it does belong here as this is the "serializer test"?
And there is already `shouldExtractValue` in `ValueTimestampHeadersDeserializerTest` -- so we can just remove this test?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers in the format [count(varint)][header1][header2]... to be processed by HeadersDeserializer.
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+ private static final HeadersDeserializer HEADERS_DESERIALIZER = new HeadersDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final Headers headers = headersDeserializer.deserialize("", rawHeaders);
Review Comment:
```suggestion
final Headers headers = headersDeserializer.deserialize(topic, rawHeaders);
```
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+
+ valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, null);
Review Comment:
```suggestion
valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
```
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
SGTM. Let's remove this method for now to get this PR merged?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers in the format [count(varint)][header1][header2]... to be processed by HeadersDeserializer.
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+ private static final HeadersDeserializer HEADERS_DESERIALIZER = new HeadersDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final Headers headers = headersDeserializer.deserialize("", rawHeaders);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.make(value, timestamp, headers);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws SerializationException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new SerializationException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract value from serialized ValueTimestampHeaders.
+ */
+ static <V> V value(final byte[] rawValueTimestampHeaders, final Deserializer<V> deserializer) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ // skip headers plus timestamp
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
+ byte[] bytes = readBytes(buffer, buffer.remaining());
+
+ return deserializer.deserialize("", bytes);
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize);
+
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
Review Comment:
```suggestion
return LONG_DESERIALIZER.deserialize("", rawTimestamp);
```
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
Review Comment:
This is the same as line 94 below -- I would remove it here?
##########
streams/src/test/java/org/apache/kafka/streams/state/ValueTimestampHeadersTest.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertTrue;
+
+public class ValueTimestampHeadersTest {
+
+ private static final String VALUE = "test-value";
+ private static final long TIMESTAMP = 123456789L;
+
+ @Test
+ public void shouldCreateInstanceWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldReturnNullWhenValueIsNullWithMake() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(null, TIMESTAMP, headers);
+
+ assertNull(valueTimestampHeaders);
+ }
+
+ @Test
+ public void shouldCreateInstanceWithMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertEquals(VALUE, valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateInstanceWithNullValueUsingMakeAllowNullable() {
+ final Headers headers = new RecordHeaders().add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(null, TIMESTAMP, headers);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNull(valueTimestampHeaders.value());
+ assertEquals(TIMESTAMP, valueTimestampHeaders.timestamp());
+ assertEquals(headers, valueTimestampHeaders.headers());
+ }
+
+ @Test
+ public void shouldCreateEmptyHeadersWhenHeadersAreNull() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ assertNotNull(valueTimestampHeaders);
+ assertNotNull(valueTimestampHeaders.headers());
+ assertEquals(0, valueTimestampHeaders.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldGetValueOrNull() {
+ final Headers headers = new RecordHeaders();
+ ValueTimestampHeaders<String> valueTimestampHeaders = ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
+
+ valueTimestampHeaders = ValueTimestampHeaders.makeAllowNullable(VALUE, TIMESTAMP, null);
+ assertEquals(VALUE, ValueTimestampHeaders.getValueOrNull(valueTimestampHeaders));
+ assertNull(ValueTimestampHeaders.getValueOrNull(null));
Review Comment:
```suggestion
assertNull(ValueTimestampHeaders.getValueOrNull(null));
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2786277415
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersDeserializer.java:
##########
@@ -86,4 +86,10 @@ public Headers deserialize(final String topic, final byte[] data) {
return headers;
}
+
+ public static Headers deserialize(final byte[] data) {
Review Comment:
>can't the caller code not just create an instance of HeaderDeserializer
what is the reason that you prefer non-static? I'm OK, just as learning purpose I ask. If the caller class call it once, it needs to create an instance and then call it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770938049
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -113,30 +127,21 @@ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
}
final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
- final int headerSize = ByteUtils.readVarint(buffer);
- buffer.position(buffer.position() + headerSize + Long.BYTES);
-
- final byte[] rawValue = new byte[buffer.remaining()];
- buffer.get(rawValue);
+ final int headersSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headersSize + Long.BYTES);
- return rawValue;
+ return readBytes(buffer, buffer.remaining());
}
/**
* Extract timestamp from serialized ValueTimestampHeaders.
*/
static long timestamp(final byte[] rawValueTimestampHeaders) {
- if (rawValueTimestampHeaders == null) {
Review Comment:
It's quite tricky here. Since `timestamp()` returns a primitive `long` (not `Long`), returning null isn't an option.
I considered throwing an `IllegalArgumentException` for `null` input to be consistent with the validation approach used in readBytes(), but that would be inconsistent with how the analogous method in `ValueAndTimestampDeserializer` handles this case.
Looking at `ValueAndTimestampDeserializer.timestamp()`, which also returns primitive long, the design choice there is to let `ByteBuffer.wrap()` throw `NullPointerException` for null input. This makes the caller responsible for null checks before invocation. I followed the same pattern for consistency with the existing codebase.
Make sense?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2790514088
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,167 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes:
+ * - For null/empty headers: headersSize = 0, headersBytes is omitted (0 bytes)
+ * - For non-empty headers: headersSize > 0, serialized headers in the format [count(varint)][header1][header2]... to be processed by HeadersDeserializer.
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+ private static final HeadersDeserializer HEADERS_DESERIALIZER = new HeadersDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final LongDeserializer timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final Headers headers = headersDeserializer.deserialize(topic, rawHeaders);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
Review Comment:
Please use `headers` to deserialize the value.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2781734521
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
Review Comment:
I suppose we should have another jira ticket to trace it as it is a an improvement
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2781869159
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
Review Comment:
it was my suggestion. To prevent multiple threads to derserialize at the same time (=multiple times of deserialization)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770430752
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
Review Comment:
I have add a test in `ValueTimestampHeadersTest`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2774332138
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,162 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headersSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = readBytes(buffer, headersSize);
+ final byte[] rawTimestamp = readBytes(buffer, Long.BYTES);
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = readBytes(buffer, buffer.remaining());
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Reads the specified number of bytes from the buffer with validation.
+ *
+ * @param buffer the ByteBuffer to read from
+ * @param length the number of bytes to read
+ * @return the byte array containing the read bytes
+ * @throws IllegalArgumentException if buffer doesn't have enough bytes
+ */
+ private static byte[] readBytes(final ByteBuffer buffer, final int length) {
+ if (buffer.remaining() < length) {
+ throw new IllegalArgumentException(
+ "Invalid ValueTimestampHeaders format: expected " + length +
+ " bytes but only " + buffer.remaining() + " bytes remaining"
+ );
+ }
+ final byte[] bytes = new byte[length];
+ buffer.get(bytes);
+ return bytes;
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
Review Comment:
Just check it with @aliehsaeedii offline, which should be fine with default scope.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "aliehsaeedii (via GitHub)" <gi...@apache.org>.
aliehsaeedii commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2770034285
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headerSize + Long.BYTES);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ return rawValue;
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ throw new IllegalArgumentException("Cannot extract timestamp from null data");
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headerSize);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
+ }
+
+ /**
+ * Extract headers from serialized ValueTimestampHeaders.
+ */
+ static Headers headers(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ throw new IllegalArgumentException("Cannot extract headers from null data");
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
Review Comment:
var name `headersSize`?
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ if (headers == null && rawHeaders != null) {
Review Comment:
wondering whether we need thread safety here? What if multiple threads see that `(headers == null && rawHeaders != null)` and they all do the derserialization here?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
Review Comment:
should we add validation to prevent BufferUnderflowException?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
Review Comment:
the same here
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,168 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ private Headers headers;
+ private final byte[] rawHeaders;
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final Headers headers) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = headers == null ? new RecordHeaders() : headers;
+ this.rawHeaders = null;
+ }
+
+ private ValueTimestampHeaders(final V value, final long timestamp, final byte[] rawHeaders) {
+ this.value = value;
+ this.timestamp = timestamp;
+ this.headers = null;
+ this.rawHeaders = rawHeaders;
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> make(final V value,
+ final long timestamp,
+ final Headers headers) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Create a new {@link ValueTimestampHeaders} instance.
+ * The provided {@code value} may be {@code null}.
+ *
+ * @param value the value (may be {@code null})
+ * @param timestamp the timestamp
+ * @param headers the headers (may be {@code null}, treated as empty)
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance
+ */
+ public static <V> ValueTimestampHeaders<V> makeAllowNullable(final V value,
+ final long timestamp,
+ final Headers headers) {
+ return new ValueTimestampHeaders<>(value, timestamp, headers);
+ }
+
+ /**
+ * Return the wrapped {@code value} of the given {@code valueTimestampHeaders} parameter
+ * if the parameter is not {@code null}.
+ *
+ * @param valueTimestampHeaders a {@link ValueTimestampHeaders} instance; can be {@code null}
+ * @param <V> the type of the value
+ * @return the wrapped {@code value} of {@code valueTimestampHeaders} if not {@code null}; otherwise {@code null}
+ */
+ public static <V> V getValueOrNull(final ValueTimestampHeaders<V> valueTimestampHeaders) {
+ return valueTimestampHeaders == null ? null : valueTimestampHeaders.value;
+ }
+
+ /**
+ * <strong>Internal use only.</strong> This method is used by the deserialization infrastructure
+ * and should not be called directly by application code.
+ * <p>
+ * Create a new {@link ValueTimestampHeaders} instance with raw (serialized) headers for lazy deserialization.
+ * The headers will be deserialized lazily when {@link #headers()} is first called, minimizing overhead
+ * during range scans when headers are not accessed.
+ * <p>
+ * This method is used internally by {@link org.apache.kafka.streams.state.internals.ValueTimestampHeadersDeserializer}.
+ *
+ * @param value the value
+ * @param timestamp the timestamp
+ * @param rawHeaders the serialized headers bytes
+ * @param <V> the type of the value
+ * @return a new {@link ValueTimestampHeaders} instance if the provided {@code value} is not {@code null};
+ * otherwise {@code null} is returned
+ */
+ public static <V> ValueTimestampHeaders<V> makeWithRawHeaders(final V value,
+ final long timestamp,
+ final byte[] rawHeaders) {
+ if (value == null) {
+ return null;
+ }
+ return new ValueTimestampHeaders<>(value, timestamp, rawHeaders);
+ }
+
+ public V value() {
+ return value;
+ }
+
+ public long timestamp() {
+ return timestamp;
+ }
+
+ public Headers headers() {
+ if (headers == null && rawHeaders != null) {
+ headers = HeadersDeserializer.deserialize(rawHeaders);
+ }
+ return headers != null ? headers : new RecordHeaders();
Review Comment:
if `headers==null`, then every call to `headers()` creates a new `RecordHeaders()` instance. This violates the expected behavior that headers() returns the same object.
```
public Headers headers() {
if (headers == null) {
if (rawHeaders != null) {
headers = HeadersDeserializer.deserialize(rawHeaders);
} else {
headers = new RecordHeaders();
}
}
return headers;
}
```
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headerSize + Long.BYTES);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ return rawValue;
+ }
+
+ /**
+ * Extract timestamp from serialized ValueTimestampHeaders.
+ */
+ static long timestamp(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ throw new IllegalArgumentException("Cannot extract timestamp from null data");
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+ buffer.position(buffer.position() + headerSize);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ return LONG_DESERIALIZER.deserialize(null, rawTimestamp);
+ }
+
+ /**
+ * Extract headers from serialized ValueTimestampHeaders.
+ */
+ static Headers headers(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ throw new IllegalArgumentException("Cannot extract headers from null data");
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(rawValueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
Review Comment:
should we add validation to prevent `BufferUnderflowException`?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
Review Comment:
the same here
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializer.java:
##########
@@ -0,0 +1,158 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableDeserializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.nio.ByteBuffer;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableDeserializer;
+
+/**
+ * Deserializer for ValueTimestampHeaders.
+ *
+ * Deserialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) to be deserialized by HeadersDeserializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value to be deserialized with the provided value deserializer
+ *
+ * This is used by KIP-1271 to deserialize values with timestamps and headers from state stores.
+ */
+class ValueTimestampHeadersDeserializer<V> implements WrappingNullableDeserializer<ValueTimestampHeaders<V>, Void, V> {
+ private static final LongDeserializer LONG_DESERIALIZER = new LongDeserializer();
+
+ public final Deserializer<V> valueDeserializer;
+ private final Deserializer<Long> timestampDeserializer;
+ private final HeadersDeserializer headersDeserializer;
+
+ ValueTimestampHeadersDeserializer(final Deserializer<V> valueDeserializer) {
+ Objects.requireNonNull(valueDeserializer);
+ this.valueDeserializer = valueDeserializer;
+ this.timestampDeserializer = new LongDeserializer();
+ this.headersDeserializer = new HeadersDeserializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueDeserializer.configure(configs, isKey);
+ timestampDeserializer.configure(configs, isKey);
+ headersDeserializer.configure(configs, isKey);
+ }
+
+ @Override
+ public ValueTimestampHeaders<V> deserialize(final String topic, final byte[] valueTimestampHeaders) {
+ if (valueTimestampHeaders == null) {
+ return null;
+ }
+
+ final ByteBuffer buffer = ByteBuffer.wrap(valueTimestampHeaders);
+ final int headerSize = ByteUtils.readVarint(buffer);
+
+ final byte[] rawHeaders = new byte[headerSize];
+ buffer.get(rawHeaders);
+
+ final byte[] rawTimestamp = new byte[Long.BYTES];
+ buffer.get(rawTimestamp);
+
+ final long timestamp = timestampDeserializer.deserialize(topic, rawTimestamp);
+
+ final byte[] rawValue = new byte[buffer.remaining()];
+ buffer.get(rawValue);
+
+ final V value = valueDeserializer.deserialize(topic, rawValue);
+
+ return ValueTimestampHeaders.makeWithRawHeaders(value, timestamp, rawHeaders);
+ }
+
+ @Override
+ public void close() {
+ valueDeserializer.close();
+ timestampDeserializer.close();
+ headersDeserializer.close();
+ }
+
+ @Override
+ public void setIfUnset(final SerdeGetter getter) {
+ // ValueTimestampHeadersDeserializer never wraps a null deserializer (or configure would throw),
+ // but it may wrap a deserializer that itself wraps a null deserializer.
+ initNullableDeserializer(valueDeserializer, getter);
+ }
+
+ /**
+ * Extract raw value bytes from serialized ValueTimestampHeaders.
+ */
+ static byte[] rawValue(final byte[] rawValueTimestampHeaders) {
+ if (rawValueTimestampHeaders == null) {
+ return null;
Review Comment:
Is that intentional that `timestamp(null)` and `headers(null)` throw IllegalArgumentException but here we return null?
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,266 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
Review Comment:
Could we verify that headers are actually deserialized lazily?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/HeadersSerializer.java:
##########
@@ -0,0 +1,88 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.nio.charset.StandardCharsets;
+
+/**
+ * Serializer for Kafka Headers.
+ * <p>
+ * Serialization format (per KIP-1271):
+ * [NumHeaders(varint)][Header1][Header2]...
+ * <p>
+ * Each header:
+ * [KeyLength(varint)][KeyBytes(UTF-8)][ValueLength(varint)][ValueBytes]
+ * <p>
+ * Note: ValueLength is -1 for null values (encoded as varint).
+ * All integers are encoded as varints (signed varint encoding).
+ * <p>
+ * This serializer produces the headersBytes portion. The headersSize prefix
+ * is added by the outer serializer (e.g., ValueTimestampHeadersSerializer).
+ * <p>
+ * This is used by KIP-1271 to serialize headers for storage in state stores.
+ */
+public class HeadersSerializer implements Serializer<Headers> {
+
+ /**
+ * Serializes headers into a byte array using varint encoding per KIP-1271.
+ * <p>
+ * The output format is [count][header1][header2]... without a size prefix.
+ * The size prefix is added by the outer serializer that uses this.
+ *
+ * @param topic topic associated with data
+ * @param headers the headers to serialize (can be null)
+ * @return the serialized byte array
+ */
+ @Override
+ public byte[] serialize(final String topic, final Headers headers) {
+ try (final ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ final DataOutputStream out = new DataOutputStream(baos)) {
+
+ final Header[] headerArray = (headers == null) ? new Header[0] : headers.toArray();
+ ByteUtils.writeVarint(headerArray.length, out);
+
+ for (final Header header : headerArray) {
+ final byte[] keyBytes = header.key().getBytes(StandardCharsets.UTF_8);
Review Comment:
What if `header.key()` returns `null`? this will throw NPE. Does the Kafka Header interface guarantee non-null keys?
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
+
+ // Format: [headersSize(varint)][headersBytes][timestamp(8)][value]
+ try (ByteArrayOutputStream baos = new ByteArrayOutputStream();
+ DataOutputStream out = new DataOutputStream(baos)) {
+
+ ByteUtils.writeVarint(rawHeaders.length, out); // headersSize
+ out.write(rawHeaders); // [count][header1][header2]...
+ out.write(rawTimestamp); // [timestamp(8)]
+ out.write(rawValue); // [value]
+
+ return baos.toByteArray();
+ } catch (IOException e) {
+ throw new RuntimeException("Failed to serialize ValueTimestampHeaders", e);
Review Comment:
Should it be more specific? `SerializationException` instead of `RuntimeException`?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2785082035
##########
streams/src/main/java/org/apache/kafka/streams/state/ValueTimestampHeaders.java:
##########
@@ -0,0 +1,177 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.streams.state.internals.HeadersDeserializer;
+
+import java.util.Objects;
+
+/**
+ * Combines a value with its timestamp and associated record headers.
+ *
+ * @param <V> the value type
+ */
+public final class ValueTimestampHeaders<V> {
+
+ private final V value;
+ private final long timestamp;
+ //visible for test
+ volatile Headers headers;
Review Comment:
But do we have code inside KS which would be multi-threaded and do this?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779406248
##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializer.java:
##########
@@ -0,0 +1,124 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.errors.SerializationException;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.utils.ByteUtils;
+import org.apache.kafka.streams.kstream.internals.WrappingNullableSerializer;
+import org.apache.kafka.streams.processor.internals.SerdeGetter;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import java.io.ByteArrayOutputStream;
+import java.io.DataOutputStream;
+import java.io.IOException;
+import java.util.Map;
+import java.util.Objects;
+
+import static org.apache.kafka.streams.kstream.internals.WrappingNullableUtils.initNullableSerializer;
+
+/**
+ * Serializer for ValueTimestampHeaders.
+ *
+ * Serialization format (per KIP-1271):
+ * [headersSize(varint)][headersBytes][timestamp(8)][value]
+ *
+ * Where:
+ * - headersSize: Size of the headersBytes section in bytes, encoded as varint
+ * - headersBytes: Serialized headers ([count(varint)][header1][header2]...) from HeadersSerializer
+ * - timestamp: 8-byte long timestamp
+ * - value: Serialized value using the provided value serializer
+ *
+ * This is used by KIP-1271 to serialize values with timestamps and headers for state stores.
+ */
+public class ValueTimestampHeadersSerializer<V> implements WrappingNullableSerializer<ValueTimestampHeaders<V>, Void, V> {
+ public final Serializer<V> valueSerializer;
+ private final Serializer<Long> timestampSerializer;
+ private final HeadersSerializer headersSerializer;
+
+ ValueTimestampHeadersSerializer(final Serializer<V> valueSerializer) {
+ Objects.requireNonNull(valueSerializer);
+ this.valueSerializer = valueSerializer;
+ this.timestampSerializer = new LongSerializer();
+ this.headersSerializer = new HeadersSerializer();
+ }
+
+ @Override
+ public void configure(final Map<String, ?> configs, final boolean isKey) {
+ valueSerializer.configure(configs, isKey);
+ timestampSerializer.configure(configs, isKey);
+ headersSerializer.configure(configs, isKey);
+ }
+
+ @Override
+ public byte[] serialize(final String topic, final ValueTimestampHeaders<V> data) {
+ if (data == null) {
+ return null;
+ }
+ return serialize(topic, data.value(), data.timestamp(), data.headers());
+ }
+
+ public byte[] serialize(final String topic, final V data, final long timestamp, final Headers headers) {
+ if (data == null) {
+ return null;
+ }
+
+ final byte[] rawValue = valueSerializer.serialize(topic, headers, data);
+
+ // Since we can't control the result of the internal serializer, we make sure that the result
+ // is not null as well.
+ // Serializing non-null values to null can be useful when working with Optional-like values
+ // where the Optional.empty case is serialized to null.
+ // See the discussion here: https://github.com/apache/kafka/pull/7679
+ if (rawValue == null) {
+ return null;
+ }
+
+ final byte[] rawHeaders = headersSerializer.serialize(topic, headers); // [count][header1][header2]...
+ final byte[] rawTimestamp = timestampSerializer.serialize(topic, timestamp);
Review Comment:
Make sense. 💯
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779464898
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
+ assertEquals(TIMESTAMP, extractedTimestamp);
+ }
+
+ @Test
+ public void shouldExtractHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final Headers extractedHeaders = ValueTimestampHeadersDeserializer.headers(serialized);
Review Comment:
`ValueTimestampHeadersDeserializerTest` already has similar test, I will delete this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779463915
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersSerializerTest.java:
##########
@@ -0,0 +1,184 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+
+public class ValueTimestampHeadersSerializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private static final long TIMESTAMP = 123456789L;
+ private static final String VALUE = "test-value";
+
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldSerializeAndDeserializeNonNullData() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals(original.value(), deserialized.value());
+ assertEquals(original.timestamp(), deserialized.timestamp());
+ assertArrayEquals(original.headers().toArray(), deserialized.headers().toArray());
+ }
+
+ @Test
+ public void shouldSerializeNullDataAsNull() {
+ final byte[] serialized = serializer.serialize(TOPIC, null);
+ assertNull(serialized);
+ }
+
+ @Test
+ public void shouldSerializeValueWithEmptyHeaders() {
+ final Headers emptyHeaders = new RecordHeaders();
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, emptyHeaders);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes())
+ .add("key3", "value3".getBytes());
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(3, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldSerializeValueWithNullHeaders() {
+ final ValueTimestampHeaders<String> valueTimestampHeaders =
+ ValueTimestampHeaders.make(VALUE, TIMESTAMP, null);
+
+ final byte[] serialized = serializer.serialize(TOPIC, valueTimestampHeaders);
+ assertNotNull(serialized);
+
+ final ValueTimestampHeaders<String> deserialized =
+ deserializer.deserialize(TOPIC, serialized);
+
+ assertEquals(VALUE, deserialized.value());
+ assertEquals(TIMESTAMP, deserialized.timestamp());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldExtractRawValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final byte[] rawValue = ValueTimestampHeadersDeserializer.rawValue(serialized);
+ assertNotNull(rawValue);
+
+ final String deserializedValue = Serdes.String().deserializer().deserialize(TOPIC, rawValue);
+ assertEquals(VALUE, deserializedValue);
+ }
+
+ @Test
+ public void shouldExtractTimestamp() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final byte[] serialized = serializer.serialize(TOPIC, VALUE, TIMESTAMP, headers);
+
+ final long extractedTimestamp = ValueTimestampHeadersDeserializer.timestamp(serialized);
Review Comment:
`ValueTimestampHeadersDeserializerTest` already has similar test, I will delete this one.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-20121: Create ValueTimestampHeaders and its serializer/deserializer [kafka]
Posted by "frankvicky (via GitHub)" <gi...@apache.org>.
frankvicky commented on code in PR #21408:
URL: https://github.com/apache/kafka/pull/21408#discussion_r2779568254
##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ValueTimestampHeadersDeserializerTest.java:
##########
@@ -0,0 +1,290 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.header.Header;
+import org.apache.kafka.common.header.Headers;
+import org.apache.kafka.common.header.internals.RecordHeaders;
+import org.apache.kafka.common.serialization.Serde;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.streams.state.ValueTimestampHeaders;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+
+import static org.junit.jupiter.api.Assertions.assertArrayEquals;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+import static org.junit.jupiter.api.Assertions.assertNotNull;
+import static org.junit.jupiter.api.Assertions.assertNull;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+public class ValueTimestampHeadersDeserializerTest {
+
+ private static final String TOPIC = "test-topic";
+ private ValueTimestampHeadersSerializer<String> serializer;
+ private ValueTimestampHeadersDeserializer<String> deserializer;
+
+ @BeforeEach
+ void setup() {
+ serializer = new ValueTimestampHeadersSerializer<>(Serdes.String().serializer());
+ deserializer = new ValueTimestampHeadersDeserializer<>(Serdes.String().deserializer());
+ }
+
+ @AfterEach
+ void cleanup() {
+ if (serializer != null) {
+ serializer.close();
+ }
+ if (deserializer != null) {
+ deserializer.close();
+ }
+ }
+
+ @Test
+ public void shouldDeserializeNull() {
+ final ValueTimestampHeaders<String> result = deserializer.deserialize(TOPIC, null);
+ assertNull(result);
+ }
+
+ @Test
+ public void shouldDeserializeWithEmptyHeaders() {
+ final Headers headers = new RecordHeaders();
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+ assertNotNull(deserialized.headers());
+ assertEquals(0, deserialized.headers().toArray().length);
+ }
+
+ @Test
+ public void shouldDeserializeWithSingleHeader() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ assertEquals(1, deserializedHeaders.toArray().length);
+
+ final Header header = deserializedHeaders.lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertArrayEquals("value1".getBytes(), header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithMultipleHeaders() {
+ final Headers headers = new RecordHeaders()
+ .add("key0", "value0".getBytes())
+ .add("key1", "value1".getBytes())
+ .add("key2", "value2".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Headers deserializedHeaders = deserialized.headers();
+ assertNotNull(deserializedHeaders);
+ Header[] headersArray = deserializedHeaders.toArray();
+ assertEquals(3, headersArray.length);
+
+ for (int i = 0; i < headersArray.length; i++) {
+ Header header = headersArray[i];
+ assertEquals("key" + i, header.key());
+ assertArrayEquals(("value" + i).getBytes(), header.value());
+ }
+ }
+
+ @Test
+ public void shouldReturnNullWhenSerializingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", "value1".getBytes());
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.makeAllowNullable(null, 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ assertNull(serialized, "Serializer should return null when value is null");
+
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+ assertNull(deserialized, "Deserializer should return null when input is null");
+ }
+
+ @Test
+ public void shouldDeserializeWithHeaderContainingNullValue() {
+ final Headers headers = new RecordHeaders()
+ .add("key1", null);
+ final ValueTimestampHeaders<String> original =
+ ValueTimestampHeaders.make("test-value", 123456789L, headers);
+
+ final byte[] serialized = serializer.serialize(TOPIC, original);
+ final ValueTimestampHeaders<String> deserialized = deserializer.deserialize(TOPIC, serialized);
+
+ assertNotNull(deserialized);
+ assertEquals("test-value", deserialized.value());
+ assertEquals(123456789L, deserialized.timestamp());
+
+ final Header header = deserialized.headers().lastHeader("key1");
+ assertNotNull(header);
+ assertEquals("key1", header.key());
+ assertNull(header.value());
+ }
+
+ @Test
+ public void shouldDeserializeWithLargeValue() {
Review Comment:
My bad, just my imagination case
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org