[GitHub] [kafka] vcrfxia opened a new pull request, #13251: KAFKA-14491: [10/N] Add changelogging wrapper for versioned stores

["vcrfxia (via GitHub)" <gi...@apache.org>]

vcrfxia opened a new pull request, #13251:
URL: https://github.com/apache/kafka/pull/13251

   (This PR is stacked on https://github.com/apache/kafka/pull/13249 and https://github.com/apache/kafka/pull/13250. Only the last commit needs to be reviewed separately.)
   
   This PR introduces the changelogging layer for the new versioned key-value store introduced in [KIP-889](https://cwiki.apache.org/confluence/display/KAFKA/KIP-889%3A+Versioned+State+Stores). We choose to have the changelogging layer operate on `VersionedBytesStore` rather than `VersionedKeyValueStore` so that the outermost store layer (metered store, see <> for more) can serialize to bytes once and then all inner stores operate only with bytes types. 
   
   ### Committer Checklist (excluded from commit message)
   - [ ] Verify design and implementation 
   - [ ] Verify test coverage and CI build status
   - [ ] Verify documentation (including upgrade notes)
   


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] mjsax merged pull request #13251: KAFKA-14491: [10/N] Add changelogging wrapper for versioned stores

["mjsax (via GitHub)" <gi...@apache.org>]

mjsax merged PR #13251:
URL: https://github.com/apache/kafka/pull/13251


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org

[GitHub] [kafka] mjsax commented on a diff in pull request #13251: KAFKA-14491: [10/N] Add changelogging wrapper for versioned stores

["mjsax (via GitHub)" <gi...@apache.org>]

mjsax commented on code in PR #13251:
URL: https://github.com/apache/kafka/pull/13251#discussion_r1109171225


##########
streams/src/main/java/org/apache/kafka/streams/state/internals/ChangeLoggingVersionedKeyValueBytesStore.java:
##########
@@ -0,0 +1,75 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.serialization.ByteArrayDeserializer;
+import org.apache.kafka.common.serialization.Deserializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.VersionedBytesStore;
+
+/**
+ * A {@link VersionedBytesStore} wrapper for writing changelog records on calls to
+ * {@link VersionedBytesStore#put(Object, Object)} and {@link VersionedBytesStore#delete(Bytes, long)}.
+ */
+public class ChangeLoggingVersionedKeyValueBytesStore extends ChangeLoggingKeyValueBytesStore implements VersionedBytesStore {
+    private static final Deserializer<ValueAndTimestamp<byte[]>> VALUE_AND_TIMESTAMP_DESERIALIZER
+        = new NullableValueAndTimestampDeserializer<>(new ByteArrayDeserializer());
+
+    private final VersionedBytesStore inner;
+
+    ChangeLoggingVersionedKeyValueBytesStore(final KeyValueStore<Bytes, byte[]> inner) {
+        super(inner);
+        if (!(inner instanceof VersionedBytesStore)) {
+            throw new IllegalStateException("inner store must be versioned");

Review Comment:
   `IllegalArgumentException` ?



##########
streams/src/test/java/org/apache/kafka/streams/state/internals/ChangeLoggingVersionedKeyValueBytesStoreTest.java:
##########
@@ -0,0 +1,227 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ *    http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import static org.hamcrest.CoreMatchers.equalTo;
+import static org.hamcrest.CoreMatchers.nullValue;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertThrows;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.verify;
+
+import java.util.Collections;
+import java.util.List;
+import org.apache.kafka.common.metrics.Metrics;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.Serializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.LogContext;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.MockStreamsMetrics;
+import org.apache.kafka.streams.state.ValueAndTimestamp;
+import org.apache.kafka.streams.state.VersionedBytesStore;
+import org.apache.kafka.test.InternalMockProcessorContext;
+import org.apache.kafka.test.MockRecordCollector;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.Before;
+import org.junit.Test;
+import org.junit.runner.RunWith;
+import org.mockito.junit.MockitoJUnitRunner;
+
+@SuppressWarnings("rawtypes")
+@RunWith(MockitoJUnitRunner.StrictStubs.class)
+public class ChangeLoggingVersionedKeyValueBytesStoreTest {
+
+    private static final Serializer<String> STRING_SERIALIZER = new StringSerializer();
+    private static final Serializer<ValueAndTimestamp<String>> VALUE_AND_TIMESTAMP_SERIALIZER
+        = new NullableValueAndTimestampSerializer<>(STRING_SERIALIZER);
+    private static final long HISTORY_RETENTION = 1000L;
+
+    private final MockRecordCollector collector = new MockRecordCollector();
+    private InternalMockProcessorContext context;
+    private VersionedBytesStore inner;
+    private ChangeLoggingVersionedKeyValueBytesStore store;
+
+    @Before
+    public void before() {
+        inner = (VersionedBytesStore) new RocksDbVersionedKeyValueBytesStoreSupplier("bytes_store", HISTORY_RETENTION).get();
+        store = new ChangeLoggingVersionedKeyValueBytesStore(inner);
+
+        context = mockContext();
+        context.setTime(0);
+        store.init((StateStoreContext) context, store);
+    }
+
+    private InternalMockProcessorContext mockContext() {
+        return new InternalMockProcessorContext<>(
+            TestUtils.tempDirectory(),
+            Serdes.String(),
+            Serdes.Long(),
+            collector,
+            new ThreadCache(new LogContext("testCache "), 0, new MockStreamsMetrics(new Metrics()))
+        );
+    }
+
+    @After
+    public void after() {
+        store.close();
+    }
+
+    @Test
+    public void shouldThrowIfInnerIsNotVersioned() {
+        assertThrows(IllegalStateException.class,
+            () -> new ChangeLoggingVersionedKeyValueBytesStore(new InMemoryKeyValueStore("kv")));
+    }
+
+    @SuppressWarnings("deprecation")
+    @Test
+    public void shouldDelegateDeprecatedInit() {
+        // recreate store with mock inner
+        store.close();
+        final VersionedBytesStore mockInner = mock(VersionedBytesStore.class);
+        store = new ChangeLoggingVersionedKeyValueBytesStore(mockInner);
+
+        store.init((ProcessorContext) context, store);
+
+        verify(mockInner).init((ProcessorContext) context, store);
+    }
+
+    @Test
+    public void shouldDelegateInit() {
+        // recreate store with mock inner
+        store.close();
+        final VersionedBytesStore mockInner = mock(VersionedBytesStore.class);
+        store = new ChangeLoggingVersionedKeyValueBytesStore(mockInner);
+
+        store.init((StateStoreContext) context, store);
+
+        verify(mockInner).init((StateStoreContext) context, store);
+    }
+
+    @Test
+    public void shouldPropagateAndLogOnPut() {
+        final Bytes rawKey = Bytes.wrap(rawBytes("k"));
+        final String value = "foo";
+        final long timestamp = 10L;
+        final byte[] rawValueAndTimestamp = rawValueAndTimestamp(value, timestamp);
+
+        store.put(rawKey, rawValueAndTimestamp);
+
+        assertThat(inner.get(rawKey), equalTo(rawValueAndTimestamp));
+        assertThat(collector.collected().size(), equalTo(1));
+        assertThat(collector.collected().get(0).key(), equalTo(rawKey));
+        assertThat(collector.collected().get(0).value(), equalTo(rawBytes(value)));
+        assertThat(collector.collected().get(0).timestamp(), equalTo(timestamp));
+    }
+
+    @Test
+    public void shouldPropagateAndLogOnPutNull() {
+        final Bytes rawKey = Bytes.wrap(rawBytes("k"));
+        final long timestamp = 10L;
+        final byte[] rawValueAndTimestamp = rawValueAndTimestamp(null, timestamp);

Review Comment:
   `rawTombstoneAndTimestamp` ?



-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org