You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/11/30 23:24:27 UTC
[GitHub] [kafka] vvcephei commented on a change in pull request #11557: KAFKA-13491: IQv2 framework
vvcephei commented on a change in pull request #11557:
URL: https://github.com/apache/kafka/pull/11557#discussion_r759708995
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/StateStoreContext.java
##########
@@ -46,8 +46,8 @@
/**
* Return the metadata of the current topic/partition/offset if available.
- * This is defined as the metadata of the record that is currently been
- * processed by the StreamTask that holds the store.
+ * This is defined as the metadata of the record that is currently being
+ * processed (or was last processed) by the StreamTask that holds the store.
Review comment:
This is a bit of a drive-by; I just noticed the wording could be improved.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/KeyValueToTimestampedKeyValueByteStoreAdapter.java
##########
@@ -115,6 +118,24 @@ public boolean isOpen() {
return store.isOpen();
}
+ @Override
+ public <R> QueryResult<R> query(
+ final Query<R> query,
+ final PositionBound positionBound,
+ final boolean collectExecutionInfo) {
+
+
+ final long start = collectExecutionInfo ? System.nanoTime() : -1L;
+ final QueryResult<R> result = store.query(query, positionBound, collectExecutionInfo);
Review comment:
Since these adapters are (presumably for a good reason) not extending WrappedStore, they have to have an implementation to delegate to their inner store.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/Stores.java
##########
@@ -122,22 +122,7 @@ public static KeyValueBytesStoreSupplier persistentTimestampedKeyValueStore(fina
*/
public static KeyValueBytesStoreSupplier inMemoryKeyValueStore(final String name) {
Objects.requireNonNull(name, "name cannot be null");
- return new KeyValueBytesStoreSupplier() {
- @Override
- public String name() {
- return name;
- }
-
- @Override
- public KeyValueStore<Bytes, byte[]> get() {
- return new InMemoryKeyValueStore(name);
- }
-
- @Override
- public String metricsScope() {
- return "in-memory";
- }
- };
+ return new InMemoryKeyValueBytesStoreSupplier(name);
Review comment:
Having this inline made it harder to debug my test, so I just popped it out into a separate class.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java
##########
@@ -118,7 +136,7 @@ private void putInternal(final Bytes key, final byte[] value) {
if (context != null && context.recordMetadata().isPresent()) {
final RecordMetadata meta = context.recordMetadata().get();
- position = position.update(meta.topic(), meta.partition(), meta.offset());
+ position = position.withComponent(meta.topic(), meta.partition(), meta.offset());
Review comment:
Just migrating to the public Position API from the KIP.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/Position.java
##########
@@ -1,96 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-package org.apache.kafka.streams.state.internals;
-
-
-import java.util.HashMap;
-import java.util.Map;
-import java.util.Map.Entry;
-import java.util.Objects;
-import java.util.concurrent.ConcurrentHashMap;
-import java.util.concurrent.ConcurrentMap;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.function.BiConsumer;
-
-public class Position {
Review comment:
Dropped the internal Position API and migrated to the new public Position API.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/StoreQueryUtils.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.query.PositionBound;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+
+public final class StoreQueryUtils {
+
+ // make this class uninstantiable
+ private StoreQueryUtils() {
+ }
+
+ @SuppressWarnings("unchecked")
+ public static <R> QueryResult<R> handleBasicQueries(
+ final Query<R> query,
+ final PositionBound positionBound,
+ final boolean collectExecutionInfo,
+ final StateStore store) {
+
+ final QueryResult<R> result;
+ final long start = collectExecutionInfo ? System.nanoTime() : -1L;
+ // TODO: position tracking
Review comment:
Just pointing out again that this isn't done in this PR.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/WrappedStateStore.java
##########
@@ -103,6 +106,22 @@ public void close() {
wrapped.close();
}
+ @Override
+ public <R> QueryResult<R> query(final Query<R> query,
Review comment:
Ok, here's the WrappedStore I keep referencing. Store layers that wrap other stores can extend this class, and if they do, they'll automatically inherit this delegating query implementation.
Nb: because of this, there are dozens of other store implementations that don't appear in this PR because they didn't require any changes.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2IntegrationTest.java
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.StreamsNotStartedException;
+import org.apache.kafka.streams.errors.StreamsStoppedException;
+import org.apache.kafka.streams.errors.UnknownStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.processor.internals.StreamThread.State;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static java.util.Collections.singleton;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+public class IQv2IntegrationTest {
Review comment:
First of two new integration tests.
This one is more focused on the framework: verifying exceptions, etc. The other one is more focused on the store/query behavior, and is therefore more interesting.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
+ GLOBAL_IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+ },
+ ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+ },
+ TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ TIME_ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedWindowStore(STORE_NAME, Duration.ofDays(1),
+ WINDOW_SIZE, false);
+ }
+ },
+ IN_MEMORY_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemorySessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ },
+ ROCKS_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentSessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ };
+
+ public abstract StoreSupplier<?> supplier();
+
+ public boolean global() {
+ return false;
+ }
+ }
+
+ @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+ public static Collection<Object[]> data() {
+ final List<Object[]> values = new ArrayList<>();
+ for (final boolean cacheEnabled : Arrays.asList(true, false)) {
+ for (final boolean logEnabled : Arrays.asList(true, false)) {
+ for (final StoresToTest toTest : StoresToTest.values()) {
Review comment:
In addition to iterating over all store types, we also iterate over caching and logging configs, so that we can be sure that IQv2 works regardless of the store configuration.
##########
File path: streams/src/test/java/org/apache/kafka/streams/state/internals/PositionTest.java
##########
@@ -1,66 +0,0 @@
-/*
- * Licensed to the Apache Software Foundation (ASF) under one or more
- * contributor license agreements. See the NOTICE file distributed with
- * this work for additional information regarding copyright ownership.
- * The ASF licenses this file to You under the Apache License, Version 2.0
- * (the "License"); you may not use this file except in compliance with
- * the License. You may obtain a copy of the License at
- *
- * http://www.apache.org/licenses/LICENSE-2.0
- *
- * Unless required by applicable law or agreed to in writing, software
- * distributed under the License is distributed on an "AS IS" BASIS,
- * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
- * See the License for the specific language governing permissions and
- * limitations under the License.
- */
-
-package org.apache.kafka.streams.state.internals;
-
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertNotEquals;
-
-import java.io.IOException;
-import org.junit.Test;
-
-public class PositionTest {
Review comment:
These cases are migrated into the new unit test.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
Review comment:
This test contains a registry of all officially provided plug-innable store types.
The test just iterates over this registry and runs the checks on each store, so this can be a good place to add tests for new queries that should be supported, and should return the same results, for all stores.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueBytesStoreSupplier.java
##########
@@ -0,0 +1,45 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+
+public class InMemoryKeyValueBytesStoreSupplier implements KeyValueBytesStoreSupplier {
Review comment:
Just extracted from Stores above.
##########
File path: streams/src/main/java/org/apache/kafka/streams/errors/StreamsStoppedException.java
##########
@@ -0,0 +1,38 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.errors;
+
+import org.apache.kafka.streams.KafkaStreams.State;
+
+/**
+ * Indicates that Kafka Streams is in a terminating or terminal state, such as {@link
+ * State#PENDING_SHUTDOWN},{@link State#PENDING_ERROR},{@link State#NOT_RUNNING}, or {@link
+ * State#ERROR}. This Streams instance will need to be discarded and replaced before it can
+ * serve queries. The caller may wish to query a different instance.
+ */
+public class StreamsStoppedException extends InvalidStateStoreException {
Review comment:
New exception proposed in the KIP. See KafkaStreams#query javadoc.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/TimestampedKeyValueStoreBuilder.java
##########
@@ -184,6 +187,20 @@ public boolean isOpen() {
return wrapped.isOpen();
}
+ @Override
+ public <R> QueryResult<R> query(final Query<R> query,
+ final PositionBound positionBound,
+ final boolean collectExecutionInfo) {
+
+ final long start = collectExecutionInfo ? System.nanoTime() : -1L;
+ final QueryResult<R> result = wrapped.query(query, positionBound, collectExecutionInfo);
Review comment:
Another explicit delegation, since this is not a WrappedStore.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/PingQuery.java
##########
@@ -0,0 +1,29 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.query.Query;
+
+/**
+ * A very simple query that all stores can handle to verify that the store is participating in the
+ * IQv2 framework properly.
+ * <p>
+ * This is not a public API and may change without notice.
+ */
+public class PingQuery implements Query<Boolean> {
Review comment:
The javadoc pretty much says it. This query is just for testing. It resides in the main module, since it needs to be visible to the store implementations, but it's not in the public API.
You'll notice that when stores handle this query, they just return `true`.
##########
File path: streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java
##########
@@ -1716,4 +1725,100 @@ protected void processStreamThread(final Consumer<StreamThread> consumer) {
return Collections.unmodifiableMap(localStorePartitionLags);
}
+
+ /**
+ * Run an interactive query against a state store.
+ * <p>
+ * This method allows callers outside of the Streams runtime to access the internal state of
+ * stateful processors. See https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html
+ * for more information.
+ *
+ * @param <R> The result type specified by the query.
+ * @throws StreamsNotStartedException If Streams has not yet been started. Just call {@link
+ * KafkaStreams#start()} and then retry this call.
+ * @throws StreamsStoppedException If Streams is in a terminal state like PENDING_SHUTDOWN,
+ * NOT_RUNNING, PENDING_ERROR, or ERROR. The caller should
+ * discover a new instance to query.
+ * @throws UnknownStateStoreException If the specified store name does not exist in the
+ * topology.
+ */
+ @Evolving
+ public <R> StateQueryResult<R> query(final StateQueryRequest<R> request) {
Review comment:
Main entry point for the new API.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2IntegrationTest.java
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.StreamsNotStartedException;
+import org.apache.kafka.streams.errors.StreamsStoppedException;
+import org.apache.kafka.streams.errors.UnknownStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.processor.internals.StreamThread.State;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static java.util.Collections.singleton;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+public class IQv2IntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreams kafkaStreams;
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+
+ builder.table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.as(STORE_NAME)
+ );
+
+ kafkaStreams = new KafkaStreams(builder.build(), streamsConfiguration());
+ kafkaStreams.cleanUp();
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void shouldFailUnknownStore() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore("unknown-store").withQuery(query);
+
+ assertThrows(UnknownStateStoreException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailNotStarted() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ assertThrows(StreamsNotStartedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailStopped() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ kafkaStreams.start();
+ kafkaStreams.close(Duration.ZERO);
+ assertThrows(StreamsStoppedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldRejectNonRunningActive()
+ throws NoSuchFieldException, IllegalAccessException {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).requireActive();
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ kafkaStreams.start();
+
+ final Field threadsField = KafkaStreams.class.getDeclaredField("threads");
+ threadsField.setAccessible(true);
+ @SuppressWarnings("unchecked") final List<StreamThread> threads =
+ (List<StreamThread>) threadsField.get(kafkaStreams);
+ final StreamThread streamThread = threads.get(0);
+
+ final Field stateLock = StreamThread.class.getDeclaredField("stateLock");
+ stateLock.setAccessible(true);
+ final Object lock = stateLock.get(streamThread);
+
+ // wait for the desired partitions to be assigned
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(
+ kafkaStreams,
+ inStore(STORE_NAME).withQuery(query),
+ partitions
+ );
+
+ // then lock the thread state, change it, and make our assertions.
+ synchronized (lock) {
+ final Field stateField = StreamThread.class.getDeclaredField("state");
+ stateField.setAccessible(true);
+ stateField.set(streamThread, State.PARTITIONS_ASSIGNED);
+
+ final StateQueryResult<Boolean> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(
+ kafkaStreams,
+ request,
+ partitions
+ );
+
+ assertThat(result.getPartitionResults().keySet(), is(partitions));
+ for (final Integer partition : partitions) {
+ assertThat(result.getPartitionResults().get(partition).isFailure(), is(true));
+ assertThat(
+ result.getPartitionResults().get(partition).getFailureReason(),
+ is(FailureReason.NOT_UP_TO_BOUND)
+ );
+ assertThat(
+ result.getPartitionResults().get(partition).getFailureMessage(),
+ is("Query requires a running active task,"
+ + " but partition was in state PARTITIONS_ASSIGNED and was active.")
+ );
+ }
+ }
+ }
+
+ @Test
+ public void shouldFetchFromPartition() {
+ final PingQuery query = new PingQuery();
+ final int partition = 1;
+ final Set<Integer> partitions = singleton(partition);
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).withPartitions(partitions);
+
+ kafkaStreams.start();
+ final StateQueryResult<Boolean> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ assertThat(result.getPartitionResults().keySet(), equalTo(partitions));
+ }
+
+ @Test
+ public void shouldFetchExplicitlyFromAllPartitions() {
+ final PingQuery query = new PingQuery();
+ final Set<Integer> partitions = mkSet(0, 1);
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).withAllPartitions();
+
+ kafkaStreams.start();
+ final StateQueryResult<Boolean> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ assertThat(result.getPartitionResults().keySet(), equalTo(partitions));
+ }
+
+ @Test
+ public void shouldNotRequireQueryHandler() {
+ final PingQuery query = new PingQuery();
+ final int partition = 1;
+ final Set<Integer> partitions = singleton(partition);
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).withPartitions(partitions);
+
+ final StreamsBuilder builder = new StreamsBuilder();
+
+ builder.table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.as(new KeyValueBytesStoreSupplier() {
+ @Override
+ public String name() {
+ return STORE_NAME;
+ }
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> get() {
+ return new KeyValueStore<Bytes, byte[]>() {
Review comment:
This takes up a lot of real estate, but what we're doing is creating a custom state store implementation that for sure doesn't implement the handler, which lets us verify the default behavior when stores don't implement the handler.
I didn't want to use any pre-existing store implementation for this, just in case we actually do decide to add a query handler to that store in the future.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2IntegrationTest.java
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.StreamsNotStartedException;
+import org.apache.kafka.streams.errors.StreamsStoppedException;
+import org.apache.kafka.streams.errors.UnknownStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.processor.internals.StreamThread.State;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static java.util.Collections.singleton;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+public class IQv2IntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreams kafkaStreams;
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+
+ builder.table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.as(STORE_NAME)
+ );
+
+ kafkaStreams = new KafkaStreams(builder.build(), streamsConfiguration());
+ kafkaStreams.cleanUp();
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void shouldFailUnknownStore() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore("unknown-store").withQuery(query);
+
+ assertThrows(UnknownStateStoreException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailNotStarted() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ assertThrows(StreamsNotStartedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailStopped() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ kafkaStreams.start();
+ kafkaStreams.close(Duration.ZERO);
+ assertThrows(StreamsStoppedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldRejectNonRunningActive()
+ throws NoSuchFieldException, IllegalAccessException {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).requireActive();
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ kafkaStreams.start();
+
+ final Field threadsField = KafkaStreams.class.getDeclaredField("threads");
+ threadsField.setAccessible(true);
+ @SuppressWarnings("unchecked") final List<StreamThread> threads =
+ (List<StreamThread>) threadsField.get(kafkaStreams);
+ final StreamThread streamThread = threads.get(0);
+
+ final Field stateLock = StreamThread.class.getDeclaredField("stateLock");
+ stateLock.setAccessible(true);
+ final Object lock = stateLock.get(streamThread);
+
+ // wait for the desired partitions to be assigned
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(
+ kafkaStreams,
+ inStore(STORE_NAME).withQuery(query),
+ partitions
+ );
+
+ // then lock the thread state, change it, and make our assertions.
+ synchronized (lock) {
Review comment:
Since we're holding this lock during the test, Streams itself will not be able to change state, which means it won't be able to surprise us by starting/ending a rebalance.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2IntegrationTest.java
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.StreamsNotStartedException;
+import org.apache.kafka.streams.errors.StreamsStoppedException;
+import org.apache.kafka.streams.errors.UnknownStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.processor.internals.StreamThread.State;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static java.util.Collections.singleton;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+public class IQv2IntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreams kafkaStreams;
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+
+ builder.table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.as(STORE_NAME)
+ );
+
+ kafkaStreams = new KafkaStreams(builder.build(), streamsConfiguration());
+ kafkaStreams.cleanUp();
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void shouldFailUnknownStore() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore("unknown-store").withQuery(query);
+
+ assertThrows(UnknownStateStoreException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailNotStarted() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ assertThrows(StreamsNotStartedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailStopped() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ kafkaStreams.start();
+ kafkaStreams.close(Duration.ZERO);
Review comment:
calling close changes the state immediately to `pending_shutdown`, so we can verify the exception without waiting for the actual close to complete.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
+ GLOBAL_IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+ },
+ ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+ },
+ TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ TIME_ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedWindowStore(STORE_NAME, Duration.ofDays(1),
+ WINDOW_SIZE, false);
+ }
+ },
+ IN_MEMORY_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemorySessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ },
+ ROCKS_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentSessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ };
+
+ public abstract StoreSupplier<?> supplier();
+
+ public boolean global() {
+ return false;
+ }
+ }
+
+ @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+ public static Collection<Object[]> data() {
+ final List<Object[]> values = new ArrayList<>();
+ for (final boolean cacheEnabled : Arrays.asList(true, false)) {
+ for (final boolean logEnabled : Arrays.asList(true, false)) {
+ for (final StoresToTest toTest : StoresToTest.values()) {
+ values.add(new Object[]{cacheEnabled, logEnabled, toTest.name()});
+ }
+ }
+ }
+ return values;
+ }
+
+ public IQv2StoreIntegrationTest(
+ final boolean cache,
+ final boolean log,
+ final String storeToTest) {
+
+ this.cache = cache;
+ this.log = log;
+ this.storeToTest = StoresToTest.valueOf(storeToTest);
+ }
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
Review comment:
For each store configuration, we will need to run a slightly different Streams application, so before each test, we create Streams and then tear it down (and clean up) afterwards.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/StoreQueryUtils.java
##########
@@ -0,0 +1,52 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.state.internals;
+
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.query.PositionBound;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+
+public final class StoreQueryUtils {
+
+ // make this class uninstantiable
+ private StoreQueryUtils() {
+ }
+
+ @SuppressWarnings("unchecked")
+ public static <R> QueryResult<R> handleBasicQueries(
Review comment:
This is the stub implementation I added to all the stores. It just returns `true` if it gets a ping query and returns an "unknown" query otherwise.
I kind of wanted to establish some other pattern for efficiently dispatching queries for other store types, but this PR was already too long, so I just kept it simple.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2IntegrationTest.java
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.errors.StreamsNotStartedException;
+import org.apache.kafka.streams.errors.StreamsStoppedException;
+import org.apache.kafka.streams.errors.UnknownStateStoreException;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.ProcessorContext;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.processor.internals.StreamThread.State;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueIterator;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+
+import java.io.IOException;
+import java.lang.reflect.Field;
+import java.time.Duration;
+import java.util.HashMap;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+
+import static java.util.Collections.singleton;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+public class IQv2IntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreams kafkaStreams;
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+
+ builder.table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ Materialized.as(STORE_NAME)
+ );
+
+ kafkaStreams = new KafkaStreams(builder.build(), streamsConfiguration());
+ kafkaStreams.cleanUp();
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void shouldFailUnknownStore() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore("unknown-store").withQuery(query);
+
+ assertThrows(UnknownStateStoreException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailNotStarted() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ assertThrows(StreamsNotStartedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldFailStopped() {
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+
+ kafkaStreams.start();
+ kafkaStreams.close(Duration.ZERO);
+ assertThrows(StreamsStoppedException.class, () -> kafkaStreams.query(request));
+ }
+
+ @Test
+ public void shouldRejectNonRunningActive()
Review comment:
Ok, this test is kind of crazy. I needed to reliably "catch" Streams when the thread hosting a task is not in RUNNING state. There's really no good way to do that, so I had two options:
1. Add a test to KafkaStreamsTest, which has an ultra-complicated mock setup, which could in theory let me drop into the right code path for this test.
2. Use reflection to manually force a non-mocked KafkaStreams into the right state to verify this behavior.
I'd obviously normally favor the mocked approach, but it really is super complicated in the case of KafkaStreamsTest. Despite being weirder, this approach is a lot simpler.
Neither approach should be more brittle than the other, since the mocks are also using reflection and will only be verified at runtime. This is probably just my confirmation bias, but I think this approach may be less brittle in practice, since we're just surgically messing with three internals instead of the dozen or so mocks and static mocks required to get a functioning KafkaStreams for the other approach.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/utils/IntegrationTestUtils.java
##########
@@ -104,6 +107,40 @@
public static final long DEFAULT_TIMEOUT = 60 * 1000L;
private static final Logger LOG = LoggerFactory.getLogger(IntegrationTestUtils.class);
+ /**
+ * Repeatedly runs the query until the response is valid and then return the response.
+ * <p>
+ * Validity in this case means that the response contains all the desired partitions or that
+ * it's a global response.
+ * <p>
+ * Once position bounding is generally supported, we should migrate tests to wait on the
+ * expected response position.
+ */
+ public static <R> StateQueryResult<R> iqv2WaitForPartitionsOrGlobal(
Review comment:
The javadoc pretty much sums this up. It's a harbinger of good things to come, since as you can see, we'll no longer be vulnerable to race conditions like rebalances sneaking in during the test. We can just wait for a valid result and then verify it.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
+ GLOBAL_IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+ },
+ ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+ },
+ TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ TIME_ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedWindowStore(STORE_NAME, Duration.ofDays(1),
+ WINDOW_SIZE, false);
+ }
+ },
+ IN_MEMORY_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemorySessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ },
+ ROCKS_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentSessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ };
+
+ public abstract StoreSupplier<?> supplier();
+
+ public boolean global() {
+ return false;
+ }
+ }
+
+ @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+ public static Collection<Object[]> data() {
+ final List<Object[]> values = new ArrayList<>();
+ for (final boolean cacheEnabled : Arrays.asList(true, false)) {
+ for (final boolean logEnabled : Arrays.asList(true, false)) {
+ for (final StoresToTest toTest : StoresToTest.values()) {
+ values.add(new Object[]{cacheEnabled, logEnabled, toTest.name()});
+ }
+ }
+ }
+ return values;
+ }
+
+ public IQv2StoreIntegrationTest(
+ final boolean cache,
+ final boolean log,
+ final String storeToTest) {
+
+ this.cache = cache;
+ this.log = log;
+ this.storeToTest = StoresToTest.valueOf(storeToTest);
+ }
+
+ @BeforeClass
Review comment:
To save time, we start Kafka and create and populate our input topic just once before all tests. None of the tests require writing new data, so this works.
##########
File path: streams/src/main/java/org/apache/kafka/streams/query/Position.java
##########
@@ -0,0 +1,172 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.query;
+
+
+import org.apache.kafka.common.annotation.InterfaceStability.Evolving;
+
+import java.util.Collections;
+import java.util.Map;
+import java.util.Map.Entry;
+import java.util.Objects;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+
+/**
+ * A representation of a position vector with respect to a set of topic partitions. For example, in
+ * Interactive Query ({@link org.apache.kafka.streams.KafkaStreams#query(StateQueryRequest)}, a
+ * query result may contain information from multiple store partitions, each of which contains
+ * information from multiple input topics' partitions. This class can be used to summarize all of
+ * that positional information.
+ * <p>
+ * This class is threadsafe, although it is mutable. Readers are recommended to use {@link
+ * Position#copy()} to avoid seeing mutations to the Position after they get the reference. For
+ * examples, when a store executes a {@link org.apache.kafka.streams.processor.StateStore#query(Query,
+ * PositionBound, boolean)} request and returns its current position via {@link
+ * QueryResult#setPosition(Position)}, it should pass a copy of its position instead of the mutable
+ * reference.
+ */
+@Evolving
+public class Position {
Review comment:
This adds the public Position API that was proposed in the KIP. I've also gone ahead and removed the private Position api (from the `state.internals` package).
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java
##########
@@ -89,6 +93,20 @@ Position getPosition() {
return position;
}
+ @Override
+ public <R> QueryResult<R> query(
+ final Query<R> query,
+ final PositionBound positionBound,
+ final boolean collectExecutionInfo) {
+
+ return StoreQueryUtils.handleBasicQueries(
+ query,
+ positionBound,
+ collectExecutionInfo,
+ this
+ );
Review comment:
I wanted to verify that the framework works correctly, but not get into adding (and testing) a real query right now, so I just added minimal query support to all stores in this PR.
I'd expect us to change these implementations to something else once kv/window/session stores start to handle different sets of queries, but for now, they all have a stub implementation that just responds to the (internal) PingQuery.
Note again: the position bound is not enforced in this PR, which is already way too long. We'll add that on (and test it) in a follow-on PR.
##########
File path: streams/src/test/java/org/apache/kafka/streams/query/PositionTest.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.query;
+
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertThrows;
+
+public class PositionTest {
+
+ @Test
+ public void shouldCreateFromMap() {
+ final Map<String, Map<Integer, Long>> map = mkMap(
+ mkEntry("topic", mkMap(mkEntry(0, 5L))),
+ mkEntry("topic1", mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ ))
+ );
+
+ final Position position = Position.fromMap(map);
+ assertThat(position.getTopics(), equalTo(mkSet("topic", "topic1")));
+ assertThat(position.getBound("topic"), equalTo(mkMap(mkEntry(0, 5L))));
+
+ // Should be a copy of the constructor map
+
+ map.get("topic1").put(99, 99L);
+
+ // so the position is still the original one
+ assertThat(position.getBound("topic1"), equalTo(mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ )));
+ }
+
+ @Test
+ public void shouldCreateFromNullMap() {
+ final Position position = Position.fromMap(null);
+ assertThat(position.getTopics(), equalTo(Collections.emptySet()));
+ }
+
+ @Test
+ public void shouldMerge() {
+ final Position position = Position.fromMap(mkMap(
+ mkEntry("topic", mkMap(mkEntry(0, 5L))),
+ mkEntry("topic1", mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ ))
+ ));
+
+ final Position position1 = Position.fromMap(mkMap(
+ mkEntry("topic", mkMap(mkEntry(0, 7L))), // update offset
+ mkEntry("topic1", mkMap(mkEntry(8, 1L))), // add partition
+ mkEntry("topic2", mkMap(mkEntry(9, 5L))) // add topic
+ ));
+
+ final Position merged = position.merge(position1);
+
+ assertThat(merged.getTopics(), equalTo(mkSet("topic", "topic1", "topic2")));
+ assertThat(merged.getBound("topic"), equalTo(mkMap(mkEntry(0, 7L))));
+ assertThat(merged.getBound("topic1"), equalTo(mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L),
+ mkEntry(8, 1L)
+ )));
+ assertThat(merged.getBound("topic2"), equalTo(mkMap(mkEntry(9, 5L))));
+ }
+
+ @Test
+ public void shouldCopy() {
+ final Position position = Position.fromMap(mkMap(
+ mkEntry("topic", mkMap(mkEntry(0, 5L))),
+ mkEntry("topic1", mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ ))
+ ));
+
+ final Position copy = position.copy();
+
+ // mutate original
+ position.withComponent("topic", 0, 6L);
+ position.withComponent("topic1", 8, 1L);
+ position.withComponent("topic2", 2, 4L);
+
+ // copy has not changed
+ assertThat(copy.getTopics(), equalTo(mkSet("topic", "topic1")));
+ assertThat(copy.getBound("topic"), equalTo(mkMap(mkEntry(0, 5L))));
+ assertThat(copy.getBound("topic1"), equalTo(mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ )));
+
+ // original has changed
+ assertThat(position.getTopics(), equalTo(mkSet("topic", "topic1", "topic2")));
+ assertThat(position.getBound("topic"), equalTo(mkMap(mkEntry(0, 6L))));
+ assertThat(position.getBound("topic1"), equalTo(mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L),
+ mkEntry(8, 1L)
+ )));
+ assertThat(position.getBound("topic2"), equalTo(mkMap(mkEntry(2, 4L))));
+ }
+
+ @Test
+ public void shouldMergeNull() {
+ final Position position = Position.fromMap(mkMap(
+ mkEntry("topic", mkMap(mkEntry(0, 5L))),
+ mkEntry("topic1", mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ ))
+ ));
+
+ final Position merged = position.merge(null);
+
+ assertThat(merged.getTopics(), equalTo(mkSet("topic", "topic1")));
+ assertThat(merged.getBound("topic"), equalTo(mkMap(mkEntry(0, 5L))));
+ assertThat(merged.getBound("topic1"), equalTo(mkMap(
+ mkEntry(0, 5L),
+ mkEntry(7, 0L)
+ )));
+ }
+
+ @Test
+ public void shouldMatchOnEqual() {
+ final Position position1 = Position.emptyPosition();
+ final Position position2 = Position.emptyPosition();
+ position1.withComponent("topic1", 0, 1);
+ position2.withComponent("topic1", 0, 1);
+
+ position1.withComponent("topic1", 1, 2);
+ position2.withComponent("topic1", 1, 2);
+
+ position1.withComponent("topic1", 2, 1);
+ position2.withComponent("topic1", 2, 1);
+
+ position1.withComponent("topic2", 0, 0);
+ position2.withComponent("topic2", 0, 0);
+
+ assertEquals(position1, position2);
+ }
+
+ @Test
+ public void shouldNotMatchOnUnEqual() {
+ final Position position1 = Position.emptyPosition();
+ final Position position2 = Position.emptyPosition();
+ position1.withComponent("topic1", 0, 1);
+ position2.withComponent("topic1", 0, 1);
+
+ position1.withComponent("topic1", 1, 2);
+
+ position1.withComponent("topic1", 2, 1);
+ position2.withComponent("topic1", 2, 1);
+
+ position1.withComponent("topic2", 0, 0);
+ position2.withComponent("topic2", 0, 0);
+
+ assertNotEquals(position1, position2);
+ }
+
+ @Test
+ public void shouldNotMatchNull() {
+ final Position position = Position.emptyPosition();
+ assertNotEquals(position, null);
+ }
+
+ @Test
+ public void shouldMatchSelf() {
+ final Position position = Position.emptyPosition();
+ assertEquals(position, position);
+ }
+
+ @Test
+ public void shouldNotHash() {
Review comment:
note: since this class is mutable, I wanted to make sure we don't accidentally use it in sets or anything.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
+ GLOBAL_IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+ },
+ ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+ },
+ TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ TIME_ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedWindowStore(STORE_NAME, Duration.ofDays(1),
+ WINDOW_SIZE, false);
+ }
+ },
+ IN_MEMORY_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemorySessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ },
+ ROCKS_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentSessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ };
+
+ public abstract StoreSupplier<?> supplier();
+
+ public boolean global() {
+ return false;
+ }
+ }
+
+ @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+ public static Collection<Object[]> data() {
+ final List<Object[]> values = new ArrayList<>();
+ for (final boolean cacheEnabled : Arrays.asList(true, false)) {
+ for (final boolean logEnabled : Arrays.asList(true, false)) {
+ for (final StoresToTest toTest : StoresToTest.values()) {
+ values.add(new Object[]{cacheEnabled, logEnabled, toTest.name()});
+ }
+ }
+ }
+ return values;
+ }
+
+ public IQv2StoreIntegrationTest(
+ final boolean cache,
+ final boolean log,
+ final String storeToTest) {
+
+ this.cache = cache;
+ this.log = log;
+ this.storeToTest = StoresToTest.valueOf(storeToTest);
+ }
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final StoreSupplier<?> supplier = storeToTest.supplier();
+ if (supplier instanceof KeyValueBytesStoreSupplier) {
+ final Materialized<Integer, Integer, KeyValueStore<Bytes, byte[]>> materialized =
+ Materialized.as((KeyValueBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (storeToTest.global()) {
+ builder
+ .globalTable(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ materialized);
+ } else {
+ builder
+ .table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ materialized);
+ }
+ } else if (supplier instanceof WindowBytesStoreSupplier) {
+ final Materialized<Integer, Integer, WindowStore<Bytes, byte[]>> materialized =
+ Materialized.as((WindowBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ builder
+ .stream(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer())
+ )
+ .groupByKey()
+ .windowedBy(TimeWindows.ofSizeWithNoGrace(WINDOW_SIZE))
+ .aggregate(
+ () -> 0,
+ (key, value, aggregate) -> aggregate + value,
+ materialized
+ );
+ } else if (supplier instanceof SessionBytesStoreSupplier) {
+ final Materialized<Integer, Integer, SessionStore<Bytes, byte[]>> materialized =
+ Materialized.as((SessionBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ builder
+ .stream(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer())
+ )
+ .groupByKey()
+ .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(WINDOW_SIZE))
+ .aggregate(
+ () -> 0,
+ (key, value, aggregate) -> aggregate + value,
+ (aggKey, aggOne, aggTwo) -> aggOne + aggTwo,
+ materialized
+ );
+ } else {
+ throw new AssertionError("Store supplier is an unrecognized type.");
+ }
+
+ // Don't need to wait for running, since tests can use iqv2 to wait until they
+ // get a valid response.
+ kafkaStreams =
+ IntegrationTestUtils.getStartedStreams(
+ streamsConfiguration(
+ cache,
+ log,
+ supplier.getClass().getSimpleName()
+ ),
+ builder,
+ true
+ );
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void verifyStore() {
+ shouldRejectUnknownQuery();
+ shouldHandlePingQuery();
+ shouldCollectExecutionInfo();
+ shouldCollectExecutionInfoUnderFailure();
+ }
Review comment:
This was a bit of a bummer for me. I was sad to learn that JUnit has no concept of Before/After Instance, so there's no way to set up the kafkaStreams application just once per store config and then run all the tests; it would have repeated the setup on each test in this class.
This is super valuable, since creating/destroying Streams can take up to a second, while running these checks just takes a few milliseconds. So we'd be spending most of our time in setup/teardown, and considering the number of parameters, it'd be a lot of time.
So, the workaround is to have just one `Test`, which calls all the checks we care about. It's slightly worse for debugging later, but the stacktrace will still tell us which check failed.
##########
File path: streams/src/test/java/org/apache/kafka/streams/query/PositionBoundTest.java
##########
@@ -0,0 +1,96 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.query;
+
+import org.junit.Test;
+
+import java.util.HashMap;
+import java.util.HashSet;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertThrows;
+import static org.junit.Assert.assertTrue;
+
+public class PositionBoundTest {
Review comment:
Just a basic unit test.
##########
File path: streams/src/main/java/org/apache/kafka/streams/state/internals/InMemoryKeyValueStore.java
##########
@@ -89,6 +93,20 @@ Position getPosition() {
return position;
}
+ @Override
+ public <R> QueryResult<R> query(
+ final Query<R> query,
+ final PositionBound positionBound,
+ final boolean collectExecutionInfo) {
+
+ return StoreQueryUtils.handleBasicQueries(
+ query,
+ positionBound,
+ collectExecutionInfo,
+ this
+ );
Review comment:
You can expect to see an identical stub in all the other provided store types.
##########
File path: streams/src/test/java/org/apache/kafka/streams/query/PositionTest.java
##########
@@ -0,0 +1,214 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+
+package org.apache.kafka.streams.query;
+
+import org.junit.Test;
+
+import java.util.Collections;
+import java.util.HashMap;
+import java.util.HashSet;
+import java.util.Map;
+
+import static org.apache.kafka.common.utils.Utils.mkEntry;
+import static org.apache.kafka.common.utils.Utils.mkMap;
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.equalTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertNotEquals;
+import static org.junit.Assert.assertThrows;
+
+public class PositionTest {
Review comment:
Another basic unit test, based on the internal API one that I removed.
##########
File path: streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java
##########
@@ -1716,4 +1725,100 @@ protected void processStreamThread(final Consumer<StreamThread> consumer) {
return Collections.unmodifiableMap(localStorePartitionLags);
}
+
+ /**
+ * Run an interactive query against a state store.
+ * <p>
+ * This method allows callers outside of the Streams runtime to access the internal state of
+ * stateful processors. See https://kafka.apache.org/documentation/streams/developer-guide/interactive-queries.html
+ * for more information.
+ *
+ * @param <R> The result type specified by the query.
+ * @throws StreamsNotStartedException If Streams has not yet been started. Just call {@link
+ * KafkaStreams#start()} and then retry this call.
+ * @throws StreamsStoppedException If Streams is in a terminal state like PENDING_SHUTDOWN,
+ * NOT_RUNNING, PENDING_ERROR, or ERROR. The caller should
+ * discover a new instance to query.
+ * @throws UnknownStateStoreException If the specified store name does not exist in the
+ * topology.
+ */
+ @Evolving
+ public <R> StateQueryResult<R> query(final StateQueryRequest<R> request) {
Review comment:
This implementation may look inefficient, but it's actually identical to the way that interactive queries are currently served. I think it could be improved, but I'm leaving it to future PRs.
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/IQv2StoreIntegrationTest.java
##########
@@ -0,0 +1,557 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.Producer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.clients.producer.RecordMetadata;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.Time;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.kstream.SessionWindows;
+import org.apache.kafka.streams.kstream.TimeWindows;
+import org.apache.kafka.streams.query.FailureReason;
+import org.apache.kafka.streams.query.Position;
+import org.apache.kafka.streams.query.Query;
+import org.apache.kafka.streams.query.QueryResult;
+import org.apache.kafka.streams.query.StateQueryRequest;
+import org.apache.kafka.streams.query.StateQueryResult;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.SessionBytesStoreSupplier;
+import org.apache.kafka.streams.state.SessionStore;
+import org.apache.kafka.streams.state.StoreSupplier;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.WindowBytesStoreSupplier;
+import org.apache.kafka.streams.state.WindowStore;
+import org.apache.kafka.streams.state.internals.PingQuery;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.runner.RunWith;
+import org.junit.runners.Parameterized;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.LinkedList;
+import java.util.List;
+import java.util.Properties;
+import java.util.Set;
+import java.util.concurrent.ExecutionException;
+import java.util.concurrent.Future;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.TimeoutException;
+import java.util.function.Consumer;
+
+import static org.apache.kafka.common.utils.Utils.mkSet;
+import static org.apache.kafka.streams.query.StateQueryRequest.inStore;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.Matchers.empty;
+import static org.hamcrest.Matchers.equalTo;
+import static org.hamcrest.Matchers.is;
+import static org.hamcrest.Matchers.matchesPattern;
+import static org.hamcrest.Matchers.not;
+import static org.hamcrest.Matchers.nullValue;
+import static org.junit.jupiter.api.Assertions.assertThrows;
+
+@Category({IntegrationTest.class})
+@RunWith(value = Parameterized.class)
+public class IQv2StoreIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ public static final Duration WINDOW_SIZE = Duration.ofMinutes(5);
+ private static int port = 0;
+ private static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final Position INPUT_POSITION = Position.emptyPosition();
+ private static final String STORE_NAME = "kv-store";
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private final StoresToTest storeToTest;
+
+ public static class UnknownQuery implements Query<Void> {
+
+ }
+
+ private final boolean cache;
+ private final boolean log;
+
+ private KafkaStreams kafkaStreams;
+
+ public enum StoresToTest {
+ GLOBAL_IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ GLOBAL_TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+
+ @Override
+ public boolean global() {
+ return true;
+ }
+ },
+ IN_MEMORY_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_LRU {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.lruMap(STORE_NAME, 100);
+ }
+ },
+ ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentKeyValueStore(STORE_NAME);
+ }
+ },
+ TIME_ROCKS_KV {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedKeyValueStore(STORE_NAME);
+ }
+ },
+ IN_MEMORY_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemoryWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentWindowStore(STORE_NAME, Duration.ofDays(1), WINDOW_SIZE,
+ false);
+ }
+ },
+ TIME_ROCKS_WINDOW {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentTimestampedWindowStore(STORE_NAME, Duration.ofDays(1),
+ WINDOW_SIZE, false);
+ }
+ },
+ IN_MEMORY_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.inMemorySessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ },
+ ROCKS_SESSION {
+ @Override
+ public StoreSupplier<?> supplier() {
+ return Stores.persistentSessionStore(STORE_NAME, Duration.ofDays(1));
+ }
+ };
+
+ public abstract StoreSupplier<?> supplier();
+
+ public boolean global() {
+ return false;
+ }
+ }
+
+ @Parameterized.Parameters(name = "cache={0}, log={1}, supplier={2}")
+ public static Collection<Object[]> data() {
+ final List<Object[]> values = new ArrayList<>();
+ for (final boolean cacheEnabled : Arrays.asList(true, false)) {
+ for (final boolean logEnabled : Arrays.asList(true, false)) {
+ for (final StoresToTest toTest : StoresToTest.values()) {
+ values.add(new Object[]{cacheEnabled, logEnabled, toTest.name()});
+ }
+ }
+ }
+ return values;
+ }
+
+ public IQv2StoreIntegrationTest(
+ final boolean cache,
+ final boolean log,
+ final String storeToTest) {
+
+ this.cache = cache;
+ this.log = log;
+ this.storeToTest = StoresToTest.valueOf(storeToTest);
+ }
+
+ @BeforeClass
+ public static void before()
+ throws InterruptedException, IOException, ExecutionException, TimeoutException {
+ CLUSTER.start();
+ final int partitions = 2;
+ CLUSTER.createTopic(INPUT_TOPIC_NAME, partitions, 1);
+
+ final Properties producerProps = new Properties();
+ producerProps.put(ProducerConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ producerProps.put(ProducerConfig.KEY_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+ producerProps.put(ProducerConfig.VALUE_SERIALIZER_CLASS_CONFIG, IntegerSerializer.class);
+
+ final List<Future<RecordMetadata>> futures = new LinkedList<>();
+ try (final Producer<Integer, Integer> producer = new KafkaProducer<>(producerProps)) {
+ for (int i = 0; i < 3; i++) {
+ final Future<RecordMetadata> send = producer.send(
+ new ProducerRecord<>(
+ INPUT_TOPIC_NAME,
+ i % partitions,
+ Time.SYSTEM.milliseconds(),
+ i,
+ i,
+ null
+ )
+ );
+ futures.add(send);
+ Time.SYSTEM.sleep(1L);
+ }
+ producer.flush();
+
+ for (final Future<RecordMetadata> future : futures) {
+ final RecordMetadata recordMetadata = future.get(1, TimeUnit.MINUTES);
+ assertThat(recordMetadata.hasOffset(), is(true));
+ INPUT_POSITION.withComponent(
+ recordMetadata.topic(),
+ recordMetadata.partition(),
+ recordMetadata.offset()
+ );
+ }
+ }
+
+ assertThat(INPUT_POSITION, equalTo(
+ Position
+ .emptyPosition()
+ .withComponent(INPUT_TOPIC_NAME, 0, 1L)
+ .withComponent(INPUT_TOPIC_NAME, 1, 0L)
+ ));
+ }
+
+ @Before
+ public void beforeTest() {
+ final StreamsBuilder builder = new StreamsBuilder();
+ final StoreSupplier<?> supplier = storeToTest.supplier();
+ if (supplier instanceof KeyValueBytesStoreSupplier) {
+ final Materialized<Integer, Integer, KeyValueStore<Bytes, byte[]>> materialized =
+ Materialized.as((KeyValueBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (storeToTest.global()) {
+ builder
+ .globalTable(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ materialized);
+ } else {
+ builder
+ .table(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer()),
+ materialized);
+ }
+ } else if (supplier instanceof WindowBytesStoreSupplier) {
+ final Materialized<Integer, Integer, WindowStore<Bytes, byte[]>> materialized =
+ Materialized.as((WindowBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ builder
+ .stream(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer())
+ )
+ .groupByKey()
+ .windowedBy(TimeWindows.ofSizeWithNoGrace(WINDOW_SIZE))
+ .aggregate(
+ () -> 0,
+ (key, value, aggregate) -> aggregate + value,
+ materialized
+ );
+ } else if (supplier instanceof SessionBytesStoreSupplier) {
+ final Materialized<Integer, Integer, SessionStore<Bytes, byte[]>> materialized =
+ Materialized.as((SessionBytesStoreSupplier) supplier);
+
+ if (cache) {
+ materialized.withCachingEnabled();
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ if (log) {
+ materialized.withLoggingEnabled(Collections.emptyMap());
+ } else {
+ materialized.withCachingDisabled();
+ }
+
+ builder
+ .stream(
+ INPUT_TOPIC_NAME,
+ Consumed.with(Serdes.Integer(), Serdes.Integer())
+ )
+ .groupByKey()
+ .windowedBy(SessionWindows.ofInactivityGapWithNoGrace(WINDOW_SIZE))
+ .aggregate(
+ () -> 0,
+ (key, value, aggregate) -> aggregate + value,
+ (aggKey, aggOne, aggTwo) -> aggOne + aggTwo,
+ materialized
+ );
+ } else {
+ throw new AssertionError("Store supplier is an unrecognized type.");
+ }
+
+ // Don't need to wait for running, since tests can use iqv2 to wait until they
+ // get a valid response.
+ kafkaStreams =
+ IntegrationTestUtils.getStartedStreams(
+ streamsConfiguration(
+ cache,
+ log,
+ supplier.getClass().getSimpleName()
+ ),
+ builder,
+ true
+ );
+ }
+
+ @After
+ public void afterTest() {
+ kafkaStreams.close();
+ kafkaStreams.cleanUp();
+ }
+
+ @AfterClass
+ public static void after() {
+ CLUSTER.stop();
+ }
+
+ @Test
+ public void verifyStore() {
+ shouldRejectUnknownQuery();
+ shouldHandlePingQuery();
+ shouldCollectExecutionInfo();
+ shouldCollectExecutionInfoUnderFailure();
+ }
+
+ public void shouldRejectUnknownQuery() {
+
+ final UnknownQuery query = new UnknownQuery();
+ final StateQueryRequest<Void> request = inStore(STORE_NAME).withQuery(query);
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ final StateQueryResult<Void> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ makeAssertions(
+ partitions,
+ result,
+ queryResult -> {
+ assertThat(queryResult.isFailure(), is(true));
+ assertThat(queryResult.isSuccess(), is(false));
+ assertThat(queryResult.getPosition(), is(nullValue()));
+ assertThat(queryResult.getFailureReason(),
+ is(FailureReason.UNKNOWN_QUERY_TYPE));
+ assertThat(queryResult.getFailureMessage(),
+ matchesPattern(
+ "This store (.*)"
+ + " doesn't know how to execute the given query"
+ + " (.*)."
+ + " Contact the store maintainer if you need support for a new query type."
+ )
+ );
+ assertThrows(IllegalArgumentException.class, queryResult::getResult);
+
+ assertThat(queryResult.getExecutionInfo(), is(empty()));
+ }
+ );
+ }
+
+ public void shouldHandlePingQuery() {
+
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query);
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ final StateQueryResult<Boolean> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ makeAssertions(
+ partitions,
+ result,
+ queryResult -> {
+ final boolean failure = queryResult.isFailure();
+ if (failure) {
+ assertThat(queryResult.toString(), failure, is(false));
+ }
+ assertThat(queryResult.isSuccess(), is(true));
+
+ // TODO: position not implemented
+ assertThat(queryResult.getPosition(), is(nullValue()));
+
+ assertThrows(IllegalArgumentException.class, queryResult::getFailureReason);
+ assertThrows(IllegalArgumentException.class, queryResult::getFailureMessage);
+
+ assertThat(queryResult.getResult(), is(true));
+
+ assertThat(queryResult.getExecutionInfo(), is(empty()));
+ });
+ }
+
+ public void shouldCollectExecutionInfo() {
+
+ final PingQuery query = new PingQuery();
+ final StateQueryRequest<Boolean> request =
+ inStore(STORE_NAME).withQuery(query).enableExecutionInfo();
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ final StateQueryResult<Boolean> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ makeAssertions(
+ partitions,
+ result,
+ queryResult -> assertThat(queryResult.getExecutionInfo(), not(empty()))
+ );
+ }
+
+ public void shouldCollectExecutionInfoUnderFailure() {
+
+ final UnknownQuery query = new UnknownQuery();
+ final StateQueryRequest<Void> request =
+ inStore(STORE_NAME).withQuery(query).enableExecutionInfo();
+ final Set<Integer> partitions = mkSet(0, 1);
+
+ final StateQueryResult<Void> result =
+ IntegrationTestUtils.iqv2WaitForPartitionsOrGlobal(kafkaStreams, request, partitions);
+
+ makeAssertions(
+ partitions,
+ result,
+ queryResult -> assertThat(queryResult.getExecutionInfo(), not(empty()))
+ );
+ }
+
+ private <R> void makeAssertions(
+ final Set<Integer> partitions,
+ final StateQueryResult<R> result,
+ final Consumer<QueryResult<R>> assertion) {
+
+ if (result.getGlobalResult() != null) {
+ assertion.accept(result.getGlobalResult());
+ } else {
+ assertThat(result.getPartitionResults().keySet(), is(partitions));
+ for (final Integer partition : partitions) {
+ assertion.accept(result.getPartitionResults().get(partition));
+ }
+ }
+ }
+
+ private static Properties streamsConfiguration(final boolean cache, final boolean log,
+ final String supplier) {
+ final String safeTestName =
+ IQv2StoreIntegrationTest.class.getName() + "-" + cache + "-" + log + "-" + supplier;
Review comment:
Note: to prevent tests from polluting each other, it's important for each test to have a unique application name and state directory.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org