You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/05/13 17:31:29 UTC
[GitHub] [kafka] jnh5y opened a new pull request, #12161: DRAFT: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
jnh5y opened a new pull request, #12161:
URL: https://github.com/apache/kafka/pull/12161
This PR adds the ability to pause and resume KafkaStreams instances as well as named/modular topologies.
Added an integration test to show how pausing and resuming works.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r886734791
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,26 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
+ final Set<TopicPartition> restoringChangelogs) {
+ // For standby tasks in the `tasks` map, we make sure that the `restoreConsumer`'s state
+ // is correct.
+ if (state == ChangelogReaderState.STANDBY_UPDATING) {
+ for (final TopicPartition partition : restoringChangelogs) {
+ final ProcessorStateManager manager = changelogs.get(partition).stateManager;
+ final TaskId taskId = manager.taskId();
+ final Task task = tasks.get(taskId);
+ if (manager.taskType() == TaskType.STANDBY) {
+ if (task != null) {
+ restoreConsumer.resume(Collections.singleton(partition));
+ } else {
+ restoreConsumer.pause(Collections.singleton(partition));
+ }
Review Comment:
I think a better way would be to collect the partitions to resume and pause and then call `restoreConsumer.pause()` and `restoreConsumer.restore()` just once with the collections of partitions.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893450731
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
Review Comment:
After some work, we've found a slightly better way to work through this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r894019831
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
Review Comment:
Thanks. I was having trouble shifting through the build info. I've updated the test to set the state store size to 0.
I'll watch the CI jobs to see if they are green after my last push. (They seemed to be ok as I was working today.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r895578928
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ }
+
Review Comment:
I'm still a little light on the details. I'll give it a try here in a little bit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893379903
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
Review Comment:
According to the builds, this test seems flaky. I think the reason is that the cache (in Streams, not the RocksDB cache) of the state stores is not set to zero. When the cache is not set to zero, the number of results that are send downstreams is not deterministic, because some intermediate results might be sent downstream and some not. I tried to set the cache to zero and was able to run 100+ times the test in a row without failure whereas with the cache > 0, the test failed much earlier.
Note that by changing the cache size also the results that you verify change.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893999817
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
Review Comment:
First, good catch! It wasn't intentional... I thought that I had found the right calls to show no lag for consumers...
I tried to switch over to verifying that there was no consumer lag, and I don't think we can sensibly do so.
(If I understand correctly) The consumer offsets are only committed after the input records have been processed. Since there is no processing, the consumer offsets will appear to be behind. (I verified this locally.)
My conclusion is that the method isn't checking anything useful, so I am planning on removing it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on PR #12161:
URL: https://github.com/apache/kafka/pull/12161#issuecomment-1156385644
>
> @jnh5y Thank you for the updates!
>
> LGTM!
>
> Had just one nit.
>
> Thank you for your patience!
@cadonna Thank you for pushing me and helping me learn more about streams!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r895651419
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
Review Comment:
I think you fixed it! 🎉
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897868472
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertTrue(kafkaStreams.allLocalStorePartitionLags().isEmpty());
+ assertTrue(kafkaStreams2.allLocalStorePartitionLags().isEmpty());
Review Comment:
Thank you! I've added these changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880913977
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1761,6 +1761,96 @@ public void shouldUpdateStandbyTask() throws Exception {
thread.taskManager().shutdown(true);
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldNotUpdateStandbyTaskWhenPaused() throws Exception {
+ final String storeName1 = "count-one";
+ final String storeName2 = "table-two";
+ final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
+ final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
+ final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
+ internalStreamsBuilder
+ .stream(Collections.singleton(topic1), consumed)
+ .groupByKey()
+ .count(Materialized.as(storeName1));
+ final MaterializedInternal<Object, Object, KeyValueStore<Bytes, byte[]>> materialized
+ = new MaterializedInternal<>(Materialized.as(storeName2), internalStreamsBuilder, "");
+ internalStreamsBuilder.table(topic2, new ConsumedInternal<>(), materialized);
+
+ internalStreamsBuilder.buildAndOptimizeTopology();
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+ restoreConsumer.updatePartitions(changelogName1,
+ Collections.singletonList(new PartitionInfo(changelogName1, 1, null, new Node[0], new Node[0]))
+ );
+
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition1, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition1, 0L));
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition2, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition2, 0L));
+ final OffsetCheckpoint checkpoint
+ = new OffsetCheckpoint(new File(stateDirectory.getOrCreateDirectoryForTask(task3), CHECKPOINT_FILE_NAME));
+ checkpoint.write(Collections.singletonMap(partition2, 5L));
+
+ thread.setState(StreamThread.State.STARTING);
+ thread.rebalanceListener().onPartitionsRevoked(Collections.emptySet());
+
+ final Map<TaskId, Set<TopicPartition>> standbyTasks = new HashMap<>();
+
+ // assign single partition
+ standbyTasks.put(task1, Collections.singleton(t1p1));
+ standbyTasks.put(task3, Collections.singleton(t2p1));
+
+ thread.taskManager().handleAssignment(emptyMap(), standbyTasks);
+ thread.taskManager().tryToCompleteRestoration(mockTime.milliseconds(), null);
+
+ thread.rebalanceListener().onPartitionsAssigned(Collections.emptyList());
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName1,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName2,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+
+ // Simulate pause
+ thread.taskManager().topologyMetadata().pauseTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // Simulate resume
+ thread.taskManager().topologyMetadata().resumeTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
Review Comment:
Good question. Short answer: I'm not certain. I copied the test here: https://github.com/apache/kafka/blob/trunk/streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java#L1684-L1762 since it is dealing with updating standby tasks.
My guess is that the method is actually returning an approximated number of entries as suggested. I tried to verify that, but I got a little lost in the unit test's complexity.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880907800
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
+
+ try {
+ streams.cleanUp();
+ fail("Should have thrown IllegalStateException");
+ } catch (final IllegalStateException expected) {
+ assertEquals("Cannot clean up while running.", expected.getMessage());
+ }
Review Comment:
I'll admit to it; I just copy-pasted the test directly above: https://github.com/apache/kafka/blob/34ec8ac54ac52196bccace8054cd7784719bf409/streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java#L816-L821
Is following the existing codebase ok or shall I spend some time to clean it up?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r886732983
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,26 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
Review Comment:
I think you forgot to also pause the active tasks in restoration.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880934812
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
+
+ final Task mockTask = createMockTask(UNNAMED_TOPOLOGY);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask, 0));
+ // This pauses an UNNAMED_TOPOLOGY / a KafkaStreams instance without named/modular
+ // topologies.
+ pausedTopologies.add(UNNAMED_TOPOLOGY);
+ Assert.assertFalse(metadata.canProcessTask(mockTask, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBePausedIndependently() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.add(TOPOLOGY1);
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBeStartedPaused() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ pausedTopologies.add(TOPOLOGY1);
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBackoff() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ metadata.registerTaskError(mockTask1, new Throwable("Error"), 0);
+ // One second after the error, task1 cannot process, task2 can.
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 1000));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 1000));
+
Review Comment:
Sure. I took a crack at updating the test; lemme know if you'd prefer different names.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r895564024
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
Review Comment:
I agree that you should remove `assertNoLag()`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897430801
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
Review Comment:
Removed!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887925475
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutor.java:
##########
@@ -275,7 +275,7 @@ private void commitSuccessfullyProcessedTasks() {
int punctuate() {
int punctuated = 0;
- for (final Task task : tasks.activeTasks()) {
+ for (final Task task : tasks.notPausedTasks()) {
Review Comment:
Note to self: Use `notPausedActiveTasks`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r888522957
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,504 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.Date;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ //properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 10000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (kafkaStreams2 != null) {
+ kafkaStreams2.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output somehow?
+ // Is there a better way to show this?
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+
+ // Start KafkaStream with paused processing.
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ // Check for rebalancing instead?
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap =
+ kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ // Create KafkaStream / NamedTopologyBuilders
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY1);
+ builder1.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY2);
+ builder2.stream(INPUT_STREAM_2)
+ .groupBy((k, v) -> k)
+ .count(IN_MEMORY_STORE)
+ .toStream()
+ .to(OUTPUT_STREAM_2);
+
+ // Start KafkaStreams instance
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause topology1
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+
+ // Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streamsNamedTopologyWrapper.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is not paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+
+ // Resume topology1 and show that it processes data.
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ // Create KafkaStreams instance / NamedTopologyBuilders
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY1);
+ builder1.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY2);
+ builder2.stream(INPUT_STREAM_2)
+ .groupBy((k, v) -> k)
+ .count(IN_MEMORY_STORE)
+ .toStream()
+ .to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams instance. This will pause all named/module topologies.
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streamsNamedTopologyWrapper.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume topology1. This will cause us to see that the instance is not paused since there
+ // is at least one topology being processed.
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2)); // Still paused
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ // Create KafkaStreams instance / NamedTopologyBuilders
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY1);
+ builder1.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streamsNamedTopologyWrapper.newNamedTopologyBuilder(TOPOLOGY2);
+ builder2.stream(INPUT_STREAM_2)
+ .groupBy((k, v) -> k)
+ .count(IN_MEMORY_STORE)
+ .toStream()
+ .to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream with topology1 paused.
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ // Write data for topology1
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams instance
+ streamsNamedTopologyWrapper.pause();
+
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streamsNamedTopologyWrapper.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is paused and has not processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume topology 1; assert that it is running.
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused()); // Are all paused? No.
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2)); // Still paused
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldWorkAcrossInstances() throws Exception {
+ // Create data on input topics (would need at least two partitions)
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Start two instances paused
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+
+ // Start KafkaStream with paused processing.
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ // Check for rebalancing instead?
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ // Create KafkaStreams instance
+ final StreamsBuilder builder2 = new StreamsBuilder();
+ builder2.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_2);
+
+ kafkaStreams2 = new KafkaStreams(builder2.build(props()), props());
+
+ // Start KafkaStream with paused processing.
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ // Check for rebalancing instead?
+ //waitForApplicationState(singletonList(kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams2.isPaused());
+
+ // Verify no data?
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // -- Verify that each instance is in rebalancing (after change to pause active task restoration)
+
+ System.out.println("JNH: calling close: " + kafkaStreams.state());
+ // Close the other -- this causes a rebalance
+ kafkaStreams2.close();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ System.out.println("JNH: called close: " + kafkaStreams.state());
+
+ // Resume paused instance
+ kafkaStreams.resume();
+ System.out.println("JNH: called resume " + kafkaStreams.state());
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ System.out.println("JNH: streams is running again " + new Date());
+
+ // Observe all data processed
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ }
Review Comment:
@cadonna I added this as a test to show what happens between multiple clients. I am noticing that this test was taking 45+ seconds to run.
It looks like it is hitting the `ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG` and I'm trying to think through why that may be happening. Any ideas?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893372025
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
Review Comment:
I think you can remove this config. It is not relevant for what this test tests.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893998039
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
Review Comment:
Yes, I think that's fair.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879798075
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
Review Comment:
Yes! I added some in this commit.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r875647755
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
Review Comment:
nit: `allNamedTopology` -> `namedTopology` since it is just one.
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
+ topologyMetadata.pauseTopology(allNamedTopology.name());
+ }
+ } else {
+ topologyMetadata.pauseTopology(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * @return true when the KafkaStreams instance has its processing paused.
+ */
+ public boolean isPaused() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ return topologyMetadata.getAllNamedTopologies()
+ .stream().map(NamedTopology::name).allMatch(topologyMetadata::isPaused);
+
+ } else {
+ return topologyMetadata.isPaused(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * This method resumes processing for the KafkaStreams instance.
+ */
+ public void resume() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
Review Comment:
nit: same as above
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
+ topologyMetadata.pauseTopology(allNamedTopology.name());
+ }
+ } else {
+ topologyMetadata.pauseTopology(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * @return true when the KafkaStreams instance has its processing paused.
+ */
+ public boolean isPaused() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ return topologyMetadata.getAllNamedTopologies()
+ .stream().map(NamedTopology::name).allMatch(topologyMetadata::isPaused);
+
Review Comment:
Please remove empty line.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadata.java:
##########
@@ -35,21 +35,27 @@ public class TaskExecutionMetadata {
private static final long CONSTANT_BACKOFF_MS = 5_000L;
private final boolean hasNamedTopologies;
+ private final Set<String> pausedTopologies;
// map of topologies experiencing errors/currently under backoff
private final ConcurrentHashMap<String, NamedTopologyMetadata> topologyNameToErrorMetadata = new ConcurrentHashMap<>();
- public TaskExecutionMetadata(final Set<String> allTopologyNames) {
+ public TaskExecutionMetadata(final Set<String> allTopologyNames, final Set<String> pausedTopologies) {
this.hasNamedTopologies = !(allTopologyNames.size() == 1 && allTopologyNames.contains(UNNAMED_TOPOLOGY));
+ this.pausedTopologies = pausedTopologies;
}
public boolean canProcessTask(final Task task, final long now) {
Review Comment:
I am aware that there are now unit tests for this class, but there are enough different code paths that would justify to add unit tests for this method.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -273,6 +274,12 @@ Collection<Task> allTasks() {
return readOnlyTasks;
}
+ Collection<Task> notPausedTasks() {
Review Comment:
Unit tests would be great!
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
+ topologyMetadata.pauseTopology(allNamedTopology.name());
+ }
+ } else {
+ topologyMetadata.pauseTopology(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * @return true when the KafkaStreams instance has its processing paused.
+ */
+ public boolean isPaused() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ return topologyMetadata.getAllNamedTopologies()
+ .stream().map(NamedTopology::name).allMatch(topologyMetadata::isPaused);
Review Comment:
```suggestion
return topologyMetadata.getAllNamedTopologies().stream()
.map(NamedTopology::name)
.allMatch(topologyMetadata::isPaused);
```
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -463,7 +463,10 @@ public void restore(final Map<TaskId, Task> tasks) {
final TaskId taskId = changelogs.get(partition).stateManager.taskId();
try {
if (restoreChangelog(changelogs.get(partition))) {
Review Comment:
Are you sure this method call avoids restoring state stores of paused tasks? Wouldn't `restoringChangelogs()` still return all changelog partitions that are in restoration and not just those that are not paused?
Maybe it is possible to add some verifications to the unit tests to ensure that only non-paused tasks are restored.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
This should also be verified in a unit test with a mock changelog reader.
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
Review Comment:
Could you please add unit tests for the new methods?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TopologyMetadata.java:
##########
@@ -257,6 +260,35 @@ public void registerAndBuildNewTopology(final KafkaFutureImpl<Void> future, fina
}
}
+ /**
+ * Pauses a topology by name
+ * @param topologyName Name of the topology to pause
+ */
+ public void pauseTopology(final String topologyName) {
Review Comment:
These methods could also be unit tested really easily.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutor.java:
##########
@@ -275,7 +275,7 @@ private void commitSuccessfullyProcessedTasks() {
int punctuate() {
int punctuated = 0;
- for (final Task task : tasks.activeTasks()) {
+ for (final Task task : tasks.notPausedTasks()) {
Review Comment:
Also here unit tests would be great and easily doable.
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
+ topologyMetadata.pauseTopology(allNamedTopology.name());
+ }
+ } else {
+ topologyMetadata.pauseTopology(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * @return true when the KafkaStreams instance has its processing paused.
+ */
+ public boolean isPaused() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ return topologyMetadata.getAllNamedTopologies()
+ .stream().map(NamedTopology::name).allMatch(topologyMetadata::isPaused);
Review Comment:
+1
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880905641
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
Review Comment:
Rolling this into one of my commits.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: DRAFT: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r872767805
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
"Yes."
The next time that a resumed task goes through the loop it will be returned in `notPausedTasks()` and `restore` will be called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a diff in pull request #12161: DRAFT: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r872731431
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -463,7 +463,10 @@ public void restore(final Map<TaskId, Task> tasks) {
final TaskId taskId = changelogs.get(partition).stateManager.taskId();
try {
if (restoreChangelog(changelogs.get(partition))) {
- tasks.get(taskId).clearTaskTimeout();
+ final Task task = tasks.get(taskId);
+ if (task != null) {
Review Comment:
Why are we getting null here when we were not before?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -273,6 +273,12 @@ Collection<Task> allTasks() {
return readOnlyTasks;
}
+ Collection<Task> notPausedTasks() {
+ return Collections.unmodifiableCollection(readOnlyActiveTasks.stream().filter(t ->
Review Comment:
nit: Can we align each call on its own line for readability?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,897 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.purgeLocalStreamsState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.startApplicationAndWaitUntilRunning;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForCompletion;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForStandbyCompletion;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.hamcrest.core.IsEqual.equalTo;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.io.File;
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import java.util.concurrent.atomic.AtomicInteger;
+import java.util.concurrent.atomic.AtomicLong;
+import org.apache.kafka.clients.admin.Admin;
+import org.apache.kafka.clients.consumer.Consumer;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.clients.consumer.KafkaConsumer;
+import org.apache.kafka.clients.producer.KafkaProducer;
+import org.apache.kafka.clients.producer.ProducerConfig;
+import org.apache.kafka.clients.producer.ProducerRecord;
+import org.apache.kafka.common.TopicPartition;
+import org.apache.kafka.common.serialization.IntegerDeserializer;
+import org.apache.kafka.common.serialization.IntegerSerializer;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.Topology;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils.TrackingStateRestoreListener;
+import org.apache.kafka.streams.kstream.Consumed;
+import org.apache.kafka.streams.kstream.KStream;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.StateRestoreListener;
+import org.apache.kafka.streams.processor.TaskId;
+import org.apache.kafka.streams.processor.api.Processor;
+import org.apache.kafka.streams.processor.api.ProcessorContext;
+import org.apache.kafka.streams.processor.api.Record;
+import org.apache.kafka.streams.processor.internals.StateDirectory;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueBytesStoreSupplier;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.streams.state.internals.InMemoryKeyValueStore;
+import org.apache.kafka.streams.state.internals.KeyValueStoreBuilder;
+import org.apache.kafka.streams.state.internals.OffsetCheckpoint;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
Review Comment:
nit: line length
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
when a paused task is resumed will restore be called if necessary?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
Review Comment:
nit: clean up?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.pause(); // This pause processing.
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
Review Comment:
I don't see where this gets cleaned up...
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
Review Comment:
you can probably just call close()
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
Review Comment:
Maybe you can wait the max poll interval?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.pause(); // This pause processing.
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streams.newNamedTopologyBuilder("topology1");
+ builder1.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streams.newNamedTopologyBuilder("topology2");
+ builder2.stream(NEW_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streams.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy1
+ // JNH: Pause topology1
+ streams.pauseNamedTopology("topology1");
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ // JNH: Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streams.isNamedTopologyPaused("topology1"));
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is not paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+
+ // Resume KafkaStreams/topology
+ // JNH: resume topology 1; assert that it is running.
+ streams.resumeNamedTopology("topology1");
+ assertFalse(streams.isNamedTopologyPaused("topoology1"));
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+ // JNH: Assert any final notes.
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streams.newNamedTopologyBuilder("topology1");
+ builder1.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streams.newNamedTopologyBuilder("topology2");
+ builder2.stream(NEW_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streams.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy1
+ // JNH: Pause topology1
+ streams.pause();
+ //streams.pauseNamedTopology("topology1");
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ // JNH: Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streams.isPaused());
+ assertTrue(streams.isNamedTopologyPaused("topology1"));
+ assertTrue(streams.isNamedTopologyPaused("topology2"));
+
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is not paused and has processed data.
+ //assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+
+ // Resume KafkaStreams/topology
+ // JNH: resume topology 1; assert that it is running.
+ streams.resumeNamedTopology("topology1");
+ assertFalse(streams.isNamedTopologyPaused("topoology1"));
+ assertFalse(streams.isPaused()); // Are all paused? No.
+ assertTrue(streams.isNamedTopologyPaused("topology2")); // Still paused
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
Review Comment:
topology2 as well
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.pause(); // This pause processing.
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streams.newNamedTopologyBuilder("topology1");
+ builder1.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streams.newNamedTopologyBuilder("topology2");
+ builder2.stream(NEW_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streams.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy1
+ // JNH: Pause topology1
+ streams.pauseNamedTopology("topology1");
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ // JNH: Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streams.isNamedTopologyPaused("topology1"));
Review Comment:
Make sure the other topology is not paused?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.pause(); // This pause processing.
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streams.newNamedTopologyBuilder("topology1");
+ builder1.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streams.newNamedTopologyBuilder("topology2");
+ builder2.stream(NEW_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streams.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy1
+ // JNH: Pause topology1
+ streams.pauseNamedTopology("topology1");
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ // JNH: Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streams.isNamedTopologyPaused("topology1"));
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is not paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+
+ // Resume KafkaStreams/topology
+ // JNH: resume topology 1; assert that it is running.
+ streams.resumeNamedTopology("topology1");
+ assertFalse(streams.isNamedTopologyPaused("topoology1"));
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+
+ // JNH: Assert any final notes.
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.deleteTopicsAndWait(NEW_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+ CLUSTER.createTopic(NEW_STREAM, 2, 1);
+
+ // Create KafkaStream / NamedTopologyBuilders
+ final KafkaStreamsNamedTopologyWrapper streams = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = streams.newNamedTopologyBuilder("topology1");
+ builder1.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ final NamedTopologyBuilder builder2 = streams.newNamedTopologyBuilder("topology2");
+ builder2.stream(NEW_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_2);
+
+ // Start KafkaStream
+ streams.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data for topology1
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy1
+ // JNH: Pause topology1
+ streams.pause();
+ //streams.pauseNamedTopology("topology1");
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ // JNH: Assert the topology1 is paused and topology2 is not paused
+ assertTrue(streams.isPaused());
+ assertTrue(streams.isNamedTopologyPaused("topology1"));
+ assertTrue(streams.isNamedTopologyPaused("topology2"));
+
+
+ // Write more data for topology 1 and topology 2
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+ produceToInputTopics(NEW_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = streams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output for topology1 somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is paused and has processed data.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_2, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Verify that topology 2 is not paused and has processed data.
Review Comment:
clean up
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
+
+ // Use statelistener to verify transition to paused transition?
+ // Read status as Paused?
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify that consumer read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ final Long maxLag = lagMap.values().stream().flatMap(m -> m.values().stream()).map(LagInfo::offsetLag).max(Long::compare).get();
+ assertEquals(0, (long) maxLag);
+
+ // Verify no output somehow? (Hard to prove negative)
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3), CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ CLUSTER.deleteTopicsAndWait(EXISTING_STREAM);
+ CLUSTER.createTopic(EXISTING_STREAM, 2, 1);
+
+ // Create KafkaStream
+ final StreamsBuilder builder = new StreamsBuilder();
+ //builder.stream(EXISTING_STREAM).groupBy((k, v) -> k).count(IN_MEMORY_STORE).toStream().to(OUTPUT_STREAM_1);
+ builder.stream(EXISTING_STREAM).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+ final Properties props = props();
+ props.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+
+ kafkaStreams = new KafkaStreams(builder.build(props), props);
+
+ // Shows pattern for blocking on state change.
+ final CountDownLatch startupLatch = new CountDownLatch(1);
+ kafkaStreams.setStateListener((newState, oldState) -> {
+ if (newState == State.RUNNING && oldState == State.REBALANCING) {
+ startupLatch.countDown();
+ }
+ });
+
+ // Start KafkaStream
+ kafkaStreams.pause(); // This pause processing.
+ kafkaStreams.start();
+ assertTrue(startupLatch.await(30, TimeUnit.SECONDS));
+
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write data
+ produceToInputTopics(EXISTING_STREAM, STANDARD_INPUT_DATA);
+
+ // Verify no output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0), CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Pause KafkaStreams/topplogy
+ kafkaStreams.pause();
Review Comment:
do we need to pause twice?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r895555390
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ }
+
Review Comment:
The comment https://github.com/apache/kafka/pull/12161/files#r893999817 is unrelated here. In my proposal, you would assert that the lag between local state store and changelog topic does not decrease. You can measure that in a similar way as you already do in `assertNoLag()`.
The restore consumer does not have a group since the partitions are manually assigned to the consumer. If there is no group there are no committed offsets.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r894001118
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
Review Comment:
Thanks! It was either going to be a good idea or prove too hacky! Glad you like it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: DRAFT: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r872767805
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
"Yes."
The next time that a resumed task goes through the loop it will be returned in `notPausedTasks()` and `restore` will be called. (I say "Yes" because that's not a separate, direct call to `restore` when resuming.)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: DRAFT: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r872766397
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -463,7 +463,10 @@ public void restore(final Map<TaskId, Task> tasks) {
final TaskId taskId = changelogs.get(partition).stateManager.taskId();
try {
if (restoreChangelog(changelogs.get(partition))) {
- tasks.get(taskId).clearTaskTimeout();
+ final Task task = tasks.get(taskId);
+ if (task != null) {
Review Comment:
The taskId is coming from the changeLogs partition info a few lines above. The tasks map returned is now smaller and does not contain all the keys anymore!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880904155
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -415,68 +415,111 @@ public Set<TopicPartition> completedChangelogs() {
public void restore(final Map<TaskId, Task> tasks) {
initializeChangelogs(tasks, registeredChangelogs());
- if (!activeRestoringChangelogs().isEmpty() && state == ChangelogReaderState.STANDBY_UPDATING) {
- throw new IllegalStateException("Should not be in standby updating state if there are still un-completed active changelogs");
+ if (!activeRestoringChangelogs().isEmpty()
+ && state == ChangelogReaderState.STANDBY_UPDATING) {
Review Comment:
Changing it back. I am still dialing in my IntelliJ settings.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879881287
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
I think I've covered this with a new test in `StreamThreadTest`.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893370030
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
Review Comment:
Do you also want to wait for `RUNNING` for the second `KafkaStreams` instance?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879881895
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadata.java:
##########
@@ -35,21 +35,27 @@ public class TaskExecutionMetadata {
private static final long CONSTANT_BACKOFF_MS = 5_000L;
private final boolean hasNamedTopologies;
+ private final Set<String> pausedTopologies;
// map of topologies experiencing errors/currently under backoff
private final ConcurrentHashMap<String, NamedTopologyMetadata> topologyNameToErrorMetadata = new ConcurrentHashMap<>();
- public TaskExecutionMetadata(final Set<String> allTopologyNames) {
+ public TaskExecutionMetadata(final Set<String> allTopologyNames, final Set<String> pausedTopologies) {
this.hasNamedTopologies = !(allTopologyNames.size() == 1 && allTopologyNames.contains(UNNAMED_TOPOLOGY));
+ this.pausedTopologies = pausedTopologies;
}
public boolean canProcessTask(final Task task, final long now) {
Review Comment:
I've added some unit tests for `TaskExecutionMetadata` here: https://github.com/apache/kafka/pull/12161/commits/0a378794226495074b0f38d5e1295ac3ec45238e
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887251426
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
+
+ final Task mockTask = createMockTask(UNNAMED_TOPOLOGY);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask, 0));
+ // This pauses an UNNAMED_TOPOLOGY / a KafkaStreams instance without named/modular
+ // topologies.
+ pausedTopologies.add(UNNAMED_TOPOLOGY);
+ Assert.assertFalse(metadata.canProcessTask(mockTask, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBePausedIndependently() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.add(TOPOLOGY1);
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBeStartedPaused() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ pausedTopologies.add(TOPOLOGY1);
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBackoff() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ metadata.registerTaskError(mockTask1, new Throwable("Error"), 0);
+ // One second after the error, task1 cannot process, task2 can.
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 1000));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 1000));
+
Review Comment:
Fixed in https://github.com/apache/kafka/pull/12161/commits/cfb18f5b61401c28d30a67ac4fbc8d0c1f12ce6f
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r895542975
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
Review Comment:
It is hacky, but still a good idea 🙂
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897869162
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
Review Comment:
My mistake; I've updated the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897893883
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,422 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("A", 3L), pair("B", 2L), pair("A", 4L), pair("C", 3L), pair("C", 4L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA_ALL = new ArrayList<KeyValue<String, Long>>() {{
+ addAll(COUNT_OUTPUT_DATA);
+ addAll(COUNT_OUTPUT_DATA2);
+ }};
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 1, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 10);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 10, COUNT_OUTPUT_DATA_ALL);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+ }
+
+ @Test
+ public void pauseResumeShouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ waitForApplicationState(singletonList(kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ final Properties properties1 = props();
+ properties1.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+ final Properties properties2 = props();
+ properties2.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1, properties1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1, properties2);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1, properties1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1, properties2);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertStreamsLagStaysConstant(kafkaStreams);
+ assertStreamsLagStaysConstant(kafkaStreams2);
Review Comment:
Changed it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897818350
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertTrue(kafkaStreams.allLocalStorePartitionLags().isEmpty());
+ assertTrue(kafkaStreams2.allLocalStorePartitionLags().isEmpty());
Review Comment:
I played a bit around with the test and indeed if you add a `Thread.sleep(2000)` before these asserts, the test fails because the returned map is not empty. That means, the assignment was not finished before the asserts were called.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887251076
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
+
+ try {
+ streams.cleanUp();
+ fail("Should have thrown IllegalStateException");
+ } catch (final IllegalStateException expected) {
+ assertEquals("Cannot clean up while running.", expected.getMessage());
+ }
Review Comment:
Fixed in https://github.com/apache/kafka/pull/12161/commits/759d3a8fd9c3a0df1ac541571c7324e67e6c002c
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887290368
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,26 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
+ final Set<TopicPartition> restoringChangelogs) {
+ // For standby tasks in the `tasks` map, we make sure that the `restoreConsumer`'s state
+ // is correct.
+ if (state == ChangelogReaderState.STANDBY_UPDATING) {
+ for (final TopicPartition partition : restoringChangelogs) {
+ final ProcessorStateManager manager = changelogs.get(partition).stateManager;
+ final TaskId taskId = manager.taskId();
+ final Task task = tasks.get(taskId);
+ if (manager.taskType() == TaskType.STANDBY) {
+ if (task != null) {
+ restoreConsumer.resume(Collections.singleton(partition));
+ } else {
+ restoreConsumer.pause(Collections.singleton(partition));
+ }
Review Comment:
Done in https://github.com/apache/kafka/pull/12161/commits/cb6d6f132a91c3373cbb6556d0b00b1d2bf8eb1b.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r894002646
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ }
+
Review Comment:
I like the idea, but based on https://github.com/apache/kafka/pull/12161/files#r893999817 I'm not sure if we can sensibly check something useful.
It seems like the offsets (at least for the main consumer) will not be committed, so they will be definitely be at 0 since processing is paused.
For the restore consumer, does it have a groupId? Would we be able to see anything about its state?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r894008227
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1686,8 +1685,74 @@ public void shouldUpdateStandbyTask() throws Exception {
final String storeName2 = "table-two";
final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+
+ setupThread(storeName1, storeName2, changelogName1, changelogName2, thread, restoreConsumer, false);
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ addStandbyRecordsToRestoreConsumer(restoreConsumer);
+
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
+
+ thread.taskManager().shutdown(true);
+ }
+
+ private void addActiveRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
+ 2,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+ }
+
+ private void addStandbyRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
Review Comment:
Extracted a constant.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879901701
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TopologyMetadata.java:
##########
@@ -257,6 +260,35 @@ public void registerAndBuildNewTopology(final KafkaFutureImpl<Void> future, fina
}
}
+ /**
+ * Pauses a topology by name
+ * @param topologyName Name of the topology to pause
+ */
+ public void pauseTopology(final String topologyName) {
Review Comment:
Added tests here: https://github.com/apache/kafka/pull/12161/commits/34ec8ac54ac52196bccace8054cd7784719bf409
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r880526257
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
Review Comment:
typo:
```suggestion
"Streams did not pause");
```
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -415,68 +415,111 @@ public Set<TopicPartition> completedChangelogs() {
public void restore(final Map<TaskId, Task> tasks) {
initializeChangelogs(tasks, registeredChangelogs());
- if (!activeRestoringChangelogs().isEmpty() && state == ChangelogReaderState.STANDBY_UPDATING) {
- throw new IllegalStateException("Should not be in standby updating state if there are still un-completed active changelogs");
+ if (!activeRestoringChangelogs().isEmpty()
+ && state == ChangelogReaderState.STANDBY_UPDATING) {
Review Comment:
nit: Why this refactoring? The condition fits in one line.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -415,68 +415,111 @@ public Set<TopicPartition> completedChangelogs() {
public void restore(final Map<TaskId, Task> tasks) {
initializeChangelogs(tasks, registeredChangelogs());
- if (!activeRestoringChangelogs().isEmpty() && state == ChangelogReaderState.STANDBY_UPDATING) {
- throw new IllegalStateException("Should not be in standby updating state if there are still un-completed active changelogs");
+ if (!activeRestoringChangelogs().isEmpty()
+ && state == ChangelogReaderState.STANDBY_UPDATING) {
+ throw new IllegalStateException(
+ "Should not be in standby updating state if there are still un-completed active changelogs");
}
if (allChangelogsCompleted()) {
log.debug("Finished restoring all changelogs {}", changelogs.keySet());
- return;
- }
+ } else {
+ final Set<TopicPartition> restoringChangelogs = restoringChangelogs();
+ if (!restoringChangelogs.isEmpty()) {
+ final ConsumerRecords<byte[], byte[]> polledRecords;
- final Set<TopicPartition> restoringChangelogs = restoringChangelogs();
- if (!restoringChangelogs.isEmpty()) {
- final ConsumerRecords<byte[], byte[]> polledRecords;
+ try {
+ // for restoring active and updating standby we may prefer different poll time
+ // in order to make sure we call the main consumer#poll in time.
+ // TODO: once we move ChangelogReader to a separate thread this may no longer be a concern
+ // JNH: Fix this?
+ // Update state based on paused/resumed status.
+ for (final TopicPartition partition : restoringChangelogs) {
+ final TaskId taskId = changelogs.get(partition).stateManager.taskId();
+ final Task task = tasks.get(taskId);
+ if (task != null) {
+ restoreConsumer.resume(Collections.singleton(partition));
+ } else {
+ restoreConsumer.pause(Collections.singleton(partition));
+ }
+ }
- try {
- // for restoring active and updating standby we may prefer different poll time
- // in order to make sure we call the main consumer#poll in time.
- // TODO: once we move ChangelogReader to a separate thread this may no longer be a concern
- polledRecords = restoreConsumer.poll(state == ChangelogReaderState.STANDBY_UPDATING ? Duration.ZERO : pollTime);
-
- // TODO (?) If we cannot fetch records during restore, should we trigger `task.timeout.ms` ?
- // TODO (?) If we cannot fetch records for standby task, should we trigger `task.timeout.ms` ?
- } catch (final InvalidOffsetException e) {
- log.warn("Encountered " + e.getClass().getName() +
- " fetching records from restore consumer for partitions " + e.partitions() + ", it is likely that " +
- "the consumer's position has fallen out of the topic partition offset range because the topic was " +
- "truncated or compacted on the broker, marking the corresponding tasks as corrupted and re-initializing" +
- " it later.", e);
-
- final Set<TaskId> corruptedTasks = new HashSet<>();
- e.partitions().forEach(partition -> corruptedTasks.add(changelogs.get(partition).stateManager.taskId()));
- throw new TaskCorruptedException(corruptedTasks, e);
- } catch (final KafkaException e) {
- throw new StreamsException("Restore consumer get unexpected error polling records.", e);
- }
+ polledRecords = restoreConsumer.poll(
+ state == ChangelogReaderState.STANDBY_UPDATING ? Duration.ZERO : pollTime);
+
+ // TODO (?) If we cannot fetch records during restore, should we trigger `task.timeout.ms` ?
+ // TODO (?) If we cannot fetch records for standby task, should we trigger `task.timeout.ms` ?
+ } catch (final InvalidOffsetException e) {
+ log.warn("Encountered " + e.getClass().getName() +
+ " fetching records from restore consumer for partitions " + e.partitions()
+ + ", it is likely that " +
+ "the consumer's position has fallen out of the topic partition offset range because the topic was "
+ +
+ "truncated or compacted on the broker, marking the corresponding tasks as corrupted and re-initializing"
+ +
+ " it later.", e);
+
+ final Set<TaskId> corruptedTasks = new HashSet<>();
+ e.partitions().forEach(partition -> corruptedTasks.add(
+ changelogs.get(partition).stateManager.taskId()));
+ throw new TaskCorruptedException(corruptedTasks, e);
+ } catch (final KafkaException e) {
+ throw new StreamsException(
+ "Restore consumer get unexpected error polling records.", e);
+ }
- for (final TopicPartition partition : polledRecords.partitions()) {
- bufferChangelogRecords(restoringChangelogByPartition(partition), polledRecords.records(partition));
- }
+ // JNH: Fix this?
Review Comment:
Please do not forget to remove those comments here and elsewhere.
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
+
+ try {
+ streams.cleanUp();
+ fail("Should have thrown IllegalStateException");
+ } catch (final IllegalStateException expected) {
+ assertEquals("Cannot clean up while running.", expected.getMessage());
+ }
Review Comment:
Wouldn't it be simpler to use `assertThrows()`?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1761,6 +1761,96 @@ public void shouldUpdateStandbyTask() throws Exception {
thread.taskManager().shutdown(true);
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldNotUpdateStandbyTaskWhenPaused() throws Exception {
+ final String storeName1 = "count-one";
+ final String storeName2 = "table-two";
+ final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
+ final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
+ final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
+ internalStreamsBuilder
+ .stream(Collections.singleton(topic1), consumed)
+ .groupByKey()
+ .count(Materialized.as(storeName1));
+ final MaterializedInternal<Object, Object, KeyValueStore<Bytes, byte[]>> materialized
+ = new MaterializedInternal<>(Materialized.as(storeName2), internalStreamsBuilder, "");
+ internalStreamsBuilder.table(topic2, new ConsumedInternal<>(), materialized);
+
+ internalStreamsBuilder.buildAndOptimizeTopology();
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+ restoreConsumer.updatePartitions(changelogName1,
+ Collections.singletonList(new PartitionInfo(changelogName1, 1, null, new Node[0], new Node[0]))
+ );
+
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition1, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition1, 0L));
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition2, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition2, 0L));
+ final OffsetCheckpoint checkpoint
+ = new OffsetCheckpoint(new File(stateDirectory.getOrCreateDirectoryForTask(task3), CHECKPOINT_FILE_NAME));
+ checkpoint.write(Collections.singletonMap(partition2, 5L));
+
+ thread.setState(StreamThread.State.STARTING);
+ thread.rebalanceListener().onPartitionsRevoked(Collections.emptySet());
+
+ final Map<TaskId, Set<TopicPartition>> standbyTasks = new HashMap<>();
+
+ // assign single partition
+ standbyTasks.put(task1, Collections.singleton(t1p1));
+ standbyTasks.put(task3, Collections.singleton(t2p1));
+
+ thread.taskManager().handleAssignment(emptyMap(), standbyTasks);
+ thread.taskManager().tryToCompleteRestoration(mockTime.milliseconds(), null);
+
+ thread.rebalanceListener().onPartitionsAssigned(Collections.emptyList());
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName1,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName2,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+
+ // Simulate pause
+ thread.taskManager().topologyMetadata().pauseTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // Simulate resume
+ thread.taskManager().topologyMetadata().resumeTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
Review Comment:
Why 4 and not 5?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -415,68 +415,111 @@ public Set<TopicPartition> completedChangelogs() {
public void restore(final Map<TaskId, Task> tasks) {
initializeChangelogs(tasks, registeredChangelogs());
- if (!activeRestoringChangelogs().isEmpty() && state == ChangelogReaderState.STANDBY_UPDATING) {
- throw new IllegalStateException("Should not be in standby updating state if there are still un-completed active changelogs");
+ if (!activeRestoringChangelogs().isEmpty()
+ && state == ChangelogReaderState.STANDBY_UPDATING) {
+ throw new IllegalStateException(
+ "Should not be in standby updating state if there are still un-completed active changelogs");
}
if (allChangelogsCompleted()) {
log.debug("Finished restoring all changelogs {}", changelogs.keySet());
- return;
- }
+ } else {
+ final Set<TopicPartition> restoringChangelogs = restoringChangelogs();
+ if (!restoringChangelogs.isEmpty()) {
+ final ConsumerRecords<byte[], byte[]> polledRecords;
- final Set<TopicPartition> restoringChangelogs = restoringChangelogs();
- if (!restoringChangelogs.isEmpty()) {
- final ConsumerRecords<byte[], byte[]> polledRecords;
+ try {
+ // for restoring active and updating standby we may prefer different poll time
+ // in order to make sure we call the main consumer#poll in time.
+ // TODO: once we move ChangelogReader to a separate thread this may no longer be a concern
Review Comment:
I think these comments should be moved before the call to `restoreConsumer.poll()`. BTW, this again confirms that inline comments are a poor way to document code, most of the times.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
+
+ final Task mockTask = createMockTask(UNNAMED_TOPOLOGY);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask, 0));
+ // This pauses an UNNAMED_TOPOLOGY / a KafkaStreams instance without named/modular
+ // topologies.
+ pausedTopologies.add(UNNAMED_TOPOLOGY);
+ Assert.assertFalse(metadata.canProcessTask(mockTask, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBePausedIndependently() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.add(TOPOLOGY1);
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBeStartedPaused() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ pausedTopologies.add(TOPOLOGY1);
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBackoff() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ metadata.registerTaskError(mockTask1, new Throwable("Error"), 0);
+ // One second after the error, task1 cannot process, task2 can.
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 1000));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 1000));
+
Review Comment:
Could you add a case where the time since the error is exactly the backoff time (i.e. 5000)?
I would prefer to put the times 1000, 5000, and 10000 into variables with meaningful names instead of adding comments.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887273657
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1761,6 +1761,96 @@ public void shouldUpdateStandbyTask() throws Exception {
thread.taskManager().shutdown(true);
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldNotUpdateStandbyTaskWhenPaused() throws Exception {
+ final String storeName1 = "count-one";
+ final String storeName2 = "table-two";
+ final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
+ final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
+ final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
+ internalStreamsBuilder
+ .stream(Collections.singleton(topic1), consumed)
+ .groupByKey()
+ .count(Materialized.as(storeName1));
+ final MaterializedInternal<Object, Object, KeyValueStore<Bytes, byte[]>> materialized
+ = new MaterializedInternal<>(Materialized.as(storeName2), internalStreamsBuilder, "");
+ internalStreamsBuilder.table(topic2, new ConsumedInternal<>(), materialized);
+
+ internalStreamsBuilder.buildAndOptimizeTopology();
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+ restoreConsumer.updatePartitions(changelogName1,
+ Collections.singletonList(new PartitionInfo(changelogName1, 1, null, new Node[0], new Node[0]))
+ );
+
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition1, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition1, 0L));
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition2, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition2, 0L));
+ final OffsetCheckpoint checkpoint
+ = new OffsetCheckpoint(new File(stateDirectory.getOrCreateDirectoryForTask(task3), CHECKPOINT_FILE_NAME));
+ checkpoint.write(Collections.singletonMap(partition2, 5L));
+
+ thread.setState(StreamThread.State.STARTING);
+ thread.rebalanceListener().onPartitionsRevoked(Collections.emptySet());
+
+ final Map<TaskId, Set<TopicPartition>> standbyTasks = new HashMap<>();
+
+ // assign single partition
+ standbyTasks.put(task1, Collections.singleton(t1p1));
+ standbyTasks.put(task3, Collections.singleton(t2p1));
+
+ thread.taskManager().handleAssignment(emptyMap(), standbyTasks);
+ thread.taskManager().tryToCompleteRestoration(mockTime.milliseconds(), null);
+
+ thread.rebalanceListener().onPartitionsAssigned(Collections.emptyList());
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName1,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName2,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+
+ // Simulate pause
+ thread.taskManager().topologyMetadata().pauseTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // Simulate resume
+ thread.taskManager().topologyMetadata().resumeTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
Review Comment:
For the refactoring, I've taken a pass at it here: https://github.com/apache/kafka/pull/12161/commits/a3bd8aec8990e2e72aec1401340c6f457db4f878.
I tend to prefer to add new things consistently with existing code, so I'm hesitating to change the check. I'm fine either way.
I'll add something for active tasks if that's the way we go.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r890648753
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
Review Comment:
Yes. I've extracted some reusable methods.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r893161440
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+ final static int TIME_ZERO = 0;
+ final static int CONSTANT_BACKOFF_MS = 5000;
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
Review Comment:
Out of curiosity, why do you use a `ConcurrentHashMap` here?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+ final static int TIME_ZERO = 0;
+ final static int CONSTANT_BACKOFF_MS = 5000;
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
Review Comment:
nit:
```suggestion
final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies, pausedTopologies);
```
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+ final static int TIME_ZERO = 0;
+ final static int CONSTANT_BACKOFF_MS = 5000;
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
+
+ final Task mockTask = createMockTask(UNNAMED_TOPOLOGY);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask, TIME_ZERO));
+ // This pauses an UNNAMED_TOPOLOGY / a KafkaStreams instance without named/modular
+ // topologies.
+ pausedTopologies.add(UNNAMED_TOPOLOGY);
+ Assert.assertFalse(metadata.canProcessTask(mockTask, TIME_ZERO));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBePausedIndependently() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
Review Comment:
nit:
```suggestion
final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES, pausedTopologies);
```
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -83,67 +137,12 @@
import org.apache.kafka.test.MockTimestampExtractor;
import org.apache.kafka.test.StreamsTestUtils;
import org.apache.kafka.test.TestUtils;
-
-import java.util.function.BiConsumer;
import org.easymock.EasyMock;
import org.junit.Assert;
import org.junit.Before;
import org.junit.Test;
import org.slf4j.Logger;
-import java.io.File;
-import java.time.Duration;
-import java.util.ArrayList;
-import java.util.Arrays;
-import java.util.Collection;
-import java.util.Collections;
-import java.util.HashMap;
-import java.util.HashSet;
-import java.util.LinkedList;
-import java.util.List;
-import java.util.Map;
-import java.util.Optional;
-import java.util.Properties;
-import java.util.Set;
-import java.util.UUID;
-import java.util.concurrent.atomic.AtomicBoolean;
-import java.util.concurrent.atomic.AtomicInteger;
-import java.util.concurrent.atomic.AtomicLong;
-import java.util.stream.Stream;
-
-import static java.util.Collections.emptyMap;
-import static java.util.Collections.emptySet;
-import static java.util.Collections.singleton;
-import static java.util.Collections.singletonMap;
-import static org.apache.kafka.common.utils.Utils.mkEntry;
-import static org.apache.kafka.common.utils.Utils.mkMap;
-import static org.apache.kafka.common.utils.Utils.mkProperties;
-import static org.apache.kafka.common.utils.Utils.mkSet;
-import static org.apache.kafka.streams.processor.internals.ClientUtils.getSharedAdminClientId;
-import static org.apache.kafka.streams.processor.internals.StateManagerUtil.CHECKPOINT_FILE_NAME;
-import static org.easymock.EasyMock.anyObject;
-import static org.easymock.EasyMock.anyInt;
-import static org.easymock.EasyMock.expect;
-import static org.easymock.EasyMock.expectLastCall;
-import static org.easymock.EasyMock.mock;
-import static org.easymock.EasyMock.niceMock;
-import static org.easymock.EasyMock.verify;
-import static org.hamcrest.CoreMatchers.equalTo;
-import static org.hamcrest.CoreMatchers.not;
-import static org.hamcrest.CoreMatchers.startsWith;
-import static org.hamcrest.MatcherAssert.assertThat;
-import static org.hamcrest.Matchers.empty;
-import static org.hamcrest.Matchers.is;
-import static org.hamcrest.Matchers.isA;
-import static org.junit.Assert.assertEquals;
-import static org.junit.Assert.assertFalse;
-import static org.junit.Assert.assertNotNull;
-import static org.junit.Assert.assertNull;
-import static org.junit.Assert.assertSame;
-import static org.junit.Assert.assertThrows;
-import static org.junit.Assert.assertTrue;
-import static org.junit.Assert.fail;
-
Review Comment:
I think you IDE reformatted the imports. The same occurred in `TaskExecutionMetadataTest`. Could you check also in other files? We use more or less the following import order:
all other imports non-static (sorted alphabetically)
<blank line>
java.* (sorted alphabetically)
javax.* (sorted alphabetically)
<blank line>
static imports (sorted alphabetically)
However, we are not really consistent across files. Nevertheless we should try to keep that order.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1686,8 +1685,74 @@ public void shouldUpdateStandbyTask() throws Exception {
final String storeName2 = "table-two";
final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+
+ setupThread(storeName1, storeName2, changelogName1, changelogName2, thread, restoreConsumer, false);
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ addStandbyRecordsToRestoreConsumer(restoreConsumer);
+
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
+
+ thread.taskManager().shutdown(true);
+ }
+
+ private void addActiveRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
+ 2,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+ }
+
+ private void addStandbyRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
Review Comment:
Why do you not pass in the store name or use a class constant? Same below and above.
##########
checkstyle/suppressions.xml:
##########
@@ -225,7 +225,7 @@
files="(EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|KTableKTableForeignKeyJoinIntegrationTest|RocksDBGenericOptionsToDbOptionsColumnFamilyOptionsAdapterTest|RelationalSmokeTest|MockProcessorContextStateStoreTest).java"/>
<suppress checks="JavaNCSS"
- files="(EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|TaskManagerTest).java"/>
+ files="(EosV2UpgradeIntegrationTest|KStreamKStreamJoinTest|StreamThreadTest|TaskManagerTest).java"/>
Review Comment:
In general, I think we should try to avoid adding suppressions. But I also see that `StreamThreadTest` would need quite some love at the moment which is not the intent of this PR.
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
Review Comment:
I like your approach!
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ }
+
Review Comment:
Could you please also add a test that verifies if active tasks in restoration and standbys are paused? Something like you start two Streams clients with 1 standby. If you have one partition one client should get the active stateful task and the other should get the standby task. The clients process some data and write some data into their states. Then shutdown the Streams clients and wipe out the local state. Finally, start both clients paused and verify if the lag of the local stores stays constant and greater than zero for a couple of poll loop iterations.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,47 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
+ final Set<TopicPartition> restoringChangelogs) {
+ if (state == ChangelogReaderState.ACTIVE_RESTORING) {
+ updatePartitionsByType(tasks, restoringChangelogs, TaskType.ACTIVE);
+ }
+ if (state == ChangelogReaderState.STANDBY_UPDATING) {
+ updatePartitionsByType(tasks, restoringChangelogs, TaskType.STANDBY);
+ }
+ }
+
+ private void updatePartitionsByType(final Map<TaskId, Task> tasks,
+ final Set<TopicPartition> restoringChangelogs,
+ final TaskType taskType) {
+ final Collection<TopicPartition> toResume =
+ restoringChangelogs.stream().filter(t -> shouldResume(tasks, t, taskType)).collect(Collectors.toList());
+ final Collection<TopicPartition> toPause =
+ restoringChangelogs.stream().filter(t -> shouldPause(tasks, t, taskType)).collect(Collectors.toList());
+ restoreConsumer.resume(toResume);
+ restoreConsumer.pause(toPause);
+ }
+
+ private boolean shouldResume(final Map<TaskId, Task> tasks, final TopicPartition partition, final TaskType taskType) {
+ final ProcessorStateManager manager = changelogs.get(partition).stateManager;
+ final TaskId taskId = manager.taskId();
+ final Task task = tasks.get(taskId);
+ if (manager.taskType() == taskType) {
+ return task != null;
+ }
+ return false;
+ }
+
+ private boolean shouldPause(final Map<TaskId, Task> tasks, final TopicPartition partition, final TaskType taskType) {
+ final ProcessorStateManager manager = changelogs.get(partition).stateManager;
+ final TaskId taskId = manager.taskId();
+ final Task task = tasks.get(taskId);
+ if (manager.taskType() == taskType) {
+ return task == null;
+ }
+ return false;
+ }
Review Comment:
Why not check the task type earlier? If the task type does not match, you do not need to do anything else.
```suggestion
final ProcessorStateManager manager = changelogs.get(partition).stateManager;
if (manager.taskType() == taskType) {
final TaskId taskId = manager.taskId();
final Task task = tasks.get(taskId);
return task == null;
}
return false;
}
```
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1686,8 +1685,74 @@ public void shouldUpdateStandbyTask() throws Exception {
final String storeName2 = "table-two";
final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+
+ setupThread(storeName1, storeName2, changelogName1, changelogName2, thread, restoreConsumer, false);
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ addStandbyRecordsToRestoreConsumer(restoreConsumer);
+
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
+
+ thread.taskManager().shutdown(true);
+ }
+
+ private void addActiveRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
+ 2,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+ }
+
+ private void addStandbyRecordsToRestoreConsumer(final MockConsumer<byte[], byte[]> restoreConsumer) {
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-count-one-changelog",
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ "stream-thread-test-table-two-changelog",
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+ }
+
+ private void setupThread(final String storeName1,
+ final String storeName2,
+ final String changelogName1,
+ final String changelogName2,
+ final StreamThread thread,
+ final MockConsumer<byte[], byte[]> restoreConsumer,
+ final boolean addActiveTask) throws IOException {
+ final TopicPartition activePartition = new TopicPartition(changelogName1, 2);
final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
Review Comment:
```suggestion
private void setupThread(final String storeName1,
final String storeName2,
final String changelogName1,
final String changelogName2,
final StreamThread thread,
final MockConsumer<byte[], byte[]> restoreConsumer,
final boolean addActiveTask) throws IOException {
final TopicPartition activePartition = new TopicPartition(changelogName1, 2);
final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
```
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,47 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
Review Comment:
After your changes that include the restoration of active tasks the name of the method does not reflect what the method does anymore. Please rename the method to something more appropriate.
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
Review Comment:
I just realized that this method computes the lag of the store regarding the changelog partition and not of the input partitions. Was this intentional? I think the lag of the store regarding the changelog topic will always be zero in this case because in normal mode the Streams client writes to the changelog topic and so it is always up-to-date.
I thought that this method verified that the main consumer does not have any lag regarding the input partitions which would tell us that the main consumer polled data although it was paused which is the expected behavior. This method with `waitUntilStreamsHasPolled(kafkaStreams, 2);` and `assertTopicSize(OUTPUT_STREAM_1, <same size as before the call to pause>);` would tell us that although the Streams clients polled data and went through the poll loop, no data was produced to the output topic. For that you should also set the cache size to 0 with `STATESTORE_CACHE_MAX_BYTES_CONFIG`.
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
Review Comment:
Typo: ` pauseResumehouldWorkAcrossInstances` -> ` pauseResumeShouldWorkAcrossInstances`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r894006082
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,126 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+ final static int TIME_ZERO = 0;
+ final static int CONSTANT_BACKOFF_MS = 5000;
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
Review Comment:
I used `ConcurrentHashMap` since that's what `pausedTopologies` is created as elsewhere.
In the unit tests, it can be a `HashSet`, I switched to that.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna merged PR #12161:
URL: https://github.com/apache/kafka/pull/12161
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897430998
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,380 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 2, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ assertNoLag(kafkaStreams);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 6);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 6);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 3, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ assertNoLag(streamsNamedTopologyWrapper);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 3);
+ assertTopicSize(OUTPUT_STREAM_2, 3);
+ }
+
+ @Test
+ public void pauseResumehouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ }
+
Review Comment:
Ok, I added a test named `pausedTopologyShouldNotRestoreStateStores` that hopefully will cover things.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897835619
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertTrue(kafkaStreams.allLocalStorePartitionLags().isEmpty());
+ assertTrue(kafkaStreams2.allLocalStorePartitionLags().isEmpty());
Review Comment:
You could do something like:
```
waitForApplicationState(Arrays.asList(kafkaStreams), State.REBALANCING, STARTUP_TIMEOUT);
waitForCondition(
() -> !kafkaStreams.allLocalStorePartitionLags().isEmpty(),
"Lags for local store partitions were not found within the timeout!");
waitUntilStreamsHasPolled(kafkaStreams, 2);
final long stateStoreLag1 = kafkaStreams.allLocalStorePartitionLags().get("test-store").get(0).offsetLag();
waitUntilStreamsHasPolled(kafkaStreams, 2);
final long stateStoreLag2 = kafkaStreams.allLocalStorePartitionLags().get("test-store").get(0).offsetLag();
assertTrue(stateStoreLag1 > 0);
assertEquals(stateStoreLag1, stateStoreLag2);
```
This code just considers one Streams client. You need to add `Materialized.as("test-store")` to the call to `count()` in your topology.
As soon as you activated the standbys, you need to do the same for the second Streams client.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897887371
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,422 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+import java.time.Duration;
+import java.util.ArrayList;
+import java.util.Arrays;
+import java.util.Collection;
+import java.util.List;
+import java.util.Properties;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.cleanStateBeforeTest;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.getTopicSize;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilStreamsHasPolled;
+import static org.apache.kafka.test.TestUtils.waitForCondition;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(1);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("A", 3L), pair("B", 2L), pair("A", 4L), pair("C", 3L), pair("C", 4L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA_ALL = new ArrayList<KeyValue<String, Long>>() {{
+ addAll(COUNT_OUTPUT_DATA);
+ addAll(COUNT_OUTPUT_DATA2);
+ }};
+
+ private String appId;
+ private KafkaStreams kafkaStreams, kafkaStreams2;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Rule
+ public final TestName testName = new TestName();
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ cleanStateBeforeTest(CLUSTER, 1, INPUT_STREAM_1, INPUT_STREAM_2, OUTPUT_STREAM_1, OUTPUT_STREAM_2);
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(ConsumerConfig.HEARTBEAT_INTERVAL_MS_CONFIG, 100);
+ properties.put(ConsumerConfig.SESSION_TIMEOUT_MS_CONFIG, 1000);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ for (final KafkaStreams streams : Arrays.asList(kafkaStreams, kafkaStreams2, streamsNamedTopologyWrapper)) {
+ if (streams != null) {
+ streams.close(Duration.ofSeconds(30));
+ }
+ }
+ }
+
+ private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(topic, records, producerConfig, CLUSTER.time);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ }
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 10);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ }
+
+ @Test
+ public void shouldPauseAndResumeAllKafkaStreamsWithNamedTopologies() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 5);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA2);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+ }
+
+ @Test
+ public void shouldAllowForNamedTopologiesToStartPaused() throws Exception {
+ streamsNamedTopologyWrapper = new KafkaStreamsNamedTopologyWrapper(props());
+ final NamedTopologyBuilder builder1 = getNamedTopologyBuilder1();
+ final NamedTopologyBuilder builder2 = getNamedTopologyBuilder2();
+
+ streamsNamedTopologyWrapper.pauseNamedTopology(TOPOLOGY1);
+ streamsNamedTopologyWrapper.start(asList(builder1.build(), builder2.build()));
+ waitForApplicationState(singletonList(streamsNamedTopologyWrapper), State.RUNNING, STARTUP_TIMEOUT);
+
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ awaitOutput(OUTPUT_STREAM_2, 5, COUNT_OUTPUT_DATA);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ streamsNamedTopologyWrapper.pause();
+ assertTrue(streamsNamedTopologyWrapper.isPaused());
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+ produceToInputTopics(INPUT_STREAM_2, STANDARD_INPUT_DATA);
+
+ waitUntilStreamsHasPolled(streamsNamedTopologyWrapper, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+
+ streamsNamedTopologyWrapper.resumeNamedTopology(TOPOLOGY1);
+ assertFalse(streamsNamedTopologyWrapper.isPaused());
+ assertFalse(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY1));
+ assertTrue(streamsNamedTopologyWrapper.isNamedTopologyPaused(TOPOLOGY2));
+
+ awaitOutput(OUTPUT_STREAM_1, 10, COUNT_OUTPUT_DATA_ALL);
+ assertTopicSize(OUTPUT_STREAM_1, 10);
+ assertTopicSize(OUTPUT_STREAM_2, 5);
+ }
+
+ @Test
+ public void pauseResumeShouldWorkAcrossInstances() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.pause();
+ kafkaStreams.start();
+
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams.isPaused());
+
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_2);
+ kafkaStreams2.pause();
+ kafkaStreams2.start();
+ waitForApplicationState(singletonList(kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+ assertTrue(kafkaStreams2.isPaused());
+
+ waitUntilStreamsHasPolled(kafkaStreams, 2);
+ waitUntilStreamsHasPolled(kafkaStreams2, 2);
+ assertTopicSize(OUTPUT_STREAM_1, 0);
+
+ kafkaStreams2.close();
+ kafkaStreams2.cleanUp();
+ waitForApplicationState(singletonList(kafkaStreams2), State.NOT_RUNNING, STARTUP_TIMEOUT);
+
+ kafkaStreams.resume();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ final Properties properties1 = props();
+ properties1.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+ final Properties properties2 = props();
+ properties2.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1, properties1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1, properties2);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1, properties1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1, properties2);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertStreamsLagStaysConstant(kafkaStreams);
+ assertStreamsLagStaysConstant(kafkaStreams2);
Review Comment:
nit: `assertStreamsLagStaysConstant()` -> `assertStreamsLocalStoreLagStaysConstant()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r897789473
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.RUNNING, STARTUP_TIMEOUT);
+
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+
+ kafkaStreams.close();
+ kafkaStreams2.close();
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams.cleanUp();
+ kafkaStreams2.cleanUp();
+
+ kafkaStreams.pause();
+ kafkaStreams2.pause();
+ kafkaStreams.start();
+ kafkaStreams2.start();
+
+ waitForApplicationState(Arrays.asList(kafkaStreams, kafkaStreams2), State.REBALANCING, STARTUP_TIMEOUT);
+
+ assertTrue(kafkaStreams.allLocalStorePartitionLags().isEmpty());
+ assertTrue(kafkaStreams2.allLocalStorePartitionLags().isEmpty());
Review Comment:
Why are you verifying for emptiness? I would expect that there are entries for the state stores with a lag greater than 0.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,47 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
Review Comment:
Do not forget to rename this method to something more meaningful.
Proposal: `pauseResumePartitions()`
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -335,7 +333,39 @@ public void pauseResumehouldWorkAcrossInstances() throws Exception {
kafkaStreams.resume();
waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
- awaitOutput(OUTPUT_STREAM_1, 3, COUNT_OUTPUT_DATA);
+ awaitOutput(OUTPUT_STREAM_1, 5, COUNT_OUTPUT_DATA);
+ }
+
+ @Test
+ public void pausedTopologyShouldNotRestoreStateStores() throws Exception {
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ kafkaStreams = buildKafkaStreams(OUTPUT_STREAM_1);
+ kafkaStreams2 = buildKafkaStreams(OUTPUT_STREAM_1);
Review Comment:
If you do not use standby tasks, there is no reason to use two Kafka Streams clients. I would propose to use one standby only for this test. For that you need to set `num.standby.replicas` to 1. That has the effect that one client gets the active store assigned and the other gets the standby store assigned.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] bvarghese1 commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
bvarghese1 commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r875206101
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ for (final NamedTopology allNamedTopology : topologyMetadata.getAllNamedTopologies()) {
+ topologyMetadata.pauseTopology(allNamedTopology.name());
+ }
+ } else {
+ topologyMetadata.pauseTopology(UNNAMED_TOPOLOGY);
+ }
+ }
+
+ /**
+ * @return true when the KafkaStreams instance has its processing paused.
+ */
+ public boolean isPaused() {
+ if (topologyMetadata.hasNamedTopologies()) {
+ return topologyMetadata.getAllNamedTopologies()
+ .stream().map(NamedTopology::name).allMatch(topologyMetadata::isPaused);
Review Comment:
Minor: Split across multiple lines for better readability. Eg: https://github.com/apache/kafka/pull/12161/files#diff-b7905c38e362cc37f6989a70b9d0cb028086fbfc79431ddbb9c64870e95482bcL230
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -273,6 +274,12 @@ Collection<Task> allTasks() {
return readOnlyTasks;
}
+ Collection<Task> notPausedTasks() {
+ return new ArrayList<>(readOnlyActiveTasks).stream().filter(t ->
Review Comment:
Minor: Split across multiple lines for better readability.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r875093015
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,440 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output somehow?
+ // Is there a better way to show this?
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+
+ // Start KafkaStream with paused processing.
+ kafkaStreams.pause();
+ kafkaStreams.start();
Review Comment:
I think the names here are a little confusing. With context, it makes sense that this is how to start a `KafkaStreams` instance with processing paused.
If anyone has a naming suggestion here, I'm very open to it!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r875817199
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,501 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.TimeUnit;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE = Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+
+ // existing topic that is pre-filled but cleared between tests
+ private static final String EXISTING_STREAM = "existing-stream";
+ private static final String NEW_STREAM = "new-stream";
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+ private static final String INPUT_STREAM_3 = "input-stream-3";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+ private static final String OUTPUT_STREAM_3 = "output-stream-3";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ // JNH: Why are there 5 outputs unlike in NamedTopologyIT?
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("A", 1L), pair("B", 1L), pair("A", 2L), pair("C", 1L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+ // output of count operation with caching
+ // private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ // asList(pair("B", 1L), pair("A", 2L), pair("C", 2L)); // output of count operation with caching
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_3, 2, 1);
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(), StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(), StringDeserializer.class, LongDeserializer.class);
+
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_3, 2, 1);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ final Properties adminConfig = new Properties();
+ adminConfig.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ }
+
+ private Properties props() {
+ final Properties streamsConfiguration = new Properties();
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ //streamsConfiguration.put(StreamsConfig.CACHE_MAX_BYTES_BUFFERING_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory(appId).getPath());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ return streamsConfiguration;
+ }
+
+ @After
+ public void shutdown() {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
Review Comment:
I'll admit that I cribbed from other tests. It looks like calling `close()` calls this method with `close(Long.MAX_VALUE)`. I think this is fine as-is.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r875834065
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -463,7 +463,10 @@ public void restore(final Map<TaskId, Task> tasks) {
final TaskId taskId = changelogs.get(partition).stateManager.taskId();
try {
if (restoreChangelog(changelogs.get(partition))) {
Review Comment:
I'll look to add a unit test to verify things.
This part may get subtle... I tried to update things and the ITs showing that one can start topologies in paused state broke.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887056404
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
Review Comment:
Ayup, that's what I understand and it is showing that the active task consumers are still reading (up until their buffers are full).
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887453359
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -273,6 +274,12 @@ Collection<Task> allTasks() {
return readOnlyTasks;
}
+ Collection<Task> notPausedTasks() {
Review Comment:
@cadonna Same deal here, I started to try and test `Tasks` instances and I started down a path of mocking quite a bit of the internal classes.
I think I'm either missing an easier approach or testing these functions directly may require a decent amount of effort.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887452460
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskExecutor.java:
##########
@@ -275,7 +275,7 @@ private void commitSuccessfullyProcessedTasks() {
int punctuate() {
int punctuated = 0;
- for (final Task task : tasks.activeTasks()) {
+ for (final Task task : tasks.notPausedTasks()) {
Review Comment:
@cadonna Ok, I tried to add some unit tests for `TaskExecutor` and I think I was needing to mock quite a few things. Does that seem right? Or is there an easier way?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r886667358
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
Review Comment:
nit:
```suggestion
private static void produceToInputTopics(final String topic, final Collection<KeyValue<String, Long>> records) {
```
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output somehow?
+ // Is there a better way to show this?
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
+
+ // Resume KafkaStreams/topology
+ kafkaStreams.resume();
+ assertFalse(kafkaStreams.isPaused());
+
+ // Verify that the input written during the pause is processed.
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA2));
+ }
+
+ @Test
+ public void shouldAllowForTopologiesToStartPaused() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+
+ // Start KafkaStream with paused processing.
+ kafkaStreams.pause();
+ kafkaStreams.start();
+ // Check for rebalancing instead?
Review Comment:
Why would you like to wait for `REBALANCING` instead of `RUNNING`?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -273,6 +274,12 @@ Collection<Task> allTasks() {
return readOnlyTasks;
}
+ Collection<Task> notPausedTasks() {
Review Comment:
Unit tests would still be great!
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
Review Comment:
This and the other comments are rather obvious. I would remove them.
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
Review Comment:
Could you please try to extract common code to reusable methods?
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
Review Comment:
Just to be clear, this works as long as the input buffers for polled records are not full. I think it is good enough for this test, though.
##########
streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java:
##########
@@ -783,6 +822,28 @@ public void shouldThrowOnCleanupWhileRunning() throws InterruptedException {
}
}
+ @Test
+ public void shouldThrowOnCleanupWhilePaused() throws InterruptedException {
+ try (final KafkaStreams streams = new KafkaStreams(getBuilderWithSource().build(), props, supplier, time)) {
+ streams.start();
+ waitForCondition(
+ () -> streams.state() == KafkaStreams.State.RUNNING,
+ "Streams never started.");
+
+ streams.pause();
+ waitForCondition(
+ streams::isPaused,
+ "Streams did not paused");
+
+ try {
+ streams.cleanUp();
+ fail("Should have thrown IllegalStateException");
+ } catch (final IllegalStateException expected) {
+ assertEquals("Cannot clean up while running.", expected.getMessage());
+ }
Review Comment:
I would use `assertThrows()` in the new test. The existing codebase also uses it in some tests like `shouldThrowExceptionSettingUncaughtExceptionHandlerNotInCreateState`.
I would open a new PR to refactor the tests that use `try-fail-catch`. That is optional.
##########
streams/src/test/java/org/apache/kafka/streams/integration/PauseResumeIntegrationTest.java:
##########
@@ -0,0 +1,445 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import static java.util.Arrays.asList;
+import static java.util.Collections.singletonList;
+import static org.apache.kafka.streams.KeyValue.pair;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.safeUniqueTestName;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitForApplicationState;
+import static org.apache.kafka.streams.integration.utils.IntegrationTestUtils.waitUntilMinKeyValueRecordsReceived;
+import static org.hamcrest.MatcherAssert.assertThat;
+import static org.junit.Assert.assertEquals;
+import static org.junit.Assert.assertFalse;
+import static org.junit.Assert.assertTrue;
+
+import java.time.Duration;
+import java.util.Collection;
+import java.util.Collections;
+import java.util.List;
+import java.util.Map;
+import java.util.Properties;
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongDeserializer;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringDeserializer;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KafkaStreams.State;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.LagInfo;
+import org.apache.kafka.streams.StreamsBuilder;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.kstream.Materialized;
+import org.apache.kafka.streams.processor.internals.namedtopology.KafkaStreamsNamedTopologyWrapper;
+import org.apache.kafka.streams.processor.internals.namedtopology.NamedTopologyBuilder;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.Stores;
+import org.apache.kafka.test.IntegrationTest;
+import org.apache.kafka.test.TestUtils;
+import org.hamcrest.CoreMatchers;
+import org.junit.After;
+import org.junit.AfterClass;
+import org.junit.Before;
+import org.junit.BeforeClass;
+import org.junit.Rule;
+import org.junit.Test;
+import org.junit.experimental.categories.Category;
+import org.junit.rules.TestName;
+
+@Category({IntegrationTest.class})
+public class PauseResumeIntegrationTest {
+ private static final Duration STARTUP_TIMEOUT = Duration.ofSeconds(45);
+ private static final int NUM_BROKERS = 1;
+
+ public static final EmbeddedKafkaCluster CLUSTER = new EmbeddedKafkaCluster(NUM_BROKERS);
+ private static Properties producerConfig;
+ private static Properties consumerConfig;
+
+ private static final Materialized<Object, Long, KeyValueStore<Bytes, byte[]>> IN_MEMORY_STORE =
+ Materialized.as(Stores.inMemoryKeyValueStore("store"));
+
+ private static final String INPUT_STREAM_1 = "input-stream-1";
+ private static final String INPUT_STREAM_2 = "input-stream-2";
+
+ private static final String OUTPUT_STREAM_1 = "output-stream-1";
+ private static final String OUTPUT_STREAM_2 = "output-stream-2";
+
+ private static final String TOPOLOGY1 = "topology1";
+ private static final String TOPOLOGY2 = "topology2";
+
+ private static final List<KeyValue<String, Long>> STANDARD_INPUT_DATA =
+ asList(pair("A", 100L), pair("B", 200L), pair("A", 300L), pair("C", 400L), pair("C", -50L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA =
+ asList(pair("B", 1L), pair("A", 2L), pair("C", 2L));
+ private static final List<KeyValue<String, Long>> COUNT_OUTPUT_DATA2 =
+ asList(pair("B", 2L), pair("A", 4L), pair("C", 4L));
+
+ @BeforeClass
+ public static void startCluster() throws Exception {
+ CLUSTER.start();
+
+ producerConfig = TestUtils.producerConfig(CLUSTER.bootstrapServers(),
+ StringSerializer.class, LongSerializer.class);
+ consumerConfig = TestUtils.consumerConfig(CLUSTER.bootstrapServers(),
+ StringDeserializer.class, LongDeserializer.class);
+ }
+
+ @AfterClass
+ public static void closeCluster() {
+ CLUSTER.stop();
+ }
+
+ @Rule
+ public final TestName testName = new TestName();
+ private String appId;
+ private KafkaStreams kafkaStreams;
+ private KafkaStreamsNamedTopologyWrapper streamsNamedTopologyWrapper;
+
+ @Before
+ public void createTopics() throws InterruptedException {
+ appId = safeUniqueTestName(PauseResumeIntegrationTest.class, testName);
+
+ CLUSTER.createTopic(INPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(INPUT_STREAM_2, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_1, 2, 1);
+ CLUSTER.createTopic(OUTPUT_STREAM_2, 2, 1);
+ }
+
+ private Properties props() {
+ final Properties properties = new Properties();
+ properties.put(StreamsConfig.APPLICATION_ID_CONFIG, appId);
+ properties.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, CLUSTER.bootstrapServers());
+ properties.put(StreamsConfig.STATE_DIR_CONFIG,
+ TestUtils.tempDirectory(appId).getPath());
+ properties.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.String().getClass());
+ properties.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.Long().getClass());
+ properties.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 1000L);
+ properties.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ properties.put(StreamsConfig.TOPOLOGY_OPTIMIZATION_CONFIG, StreamsConfig.OPTIMIZE);
+ return properties;
+ }
+
+ @After
+ public void shutdown() throws InterruptedException {
+ if (kafkaStreams != null) {
+ kafkaStreams.close(Duration.ofSeconds(30));
+ }
+ if (streamsNamedTopologyWrapper != null) {
+ streamsNamedTopologyWrapper.close(Duration.ofSeconds(30));
+ }
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(INPUT_STREAM_2);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_1);
+ CLUSTER.deleteTopicsAndWait(OUTPUT_STREAM_2);
+ }
+
+ private static void produceToInputTopics(final String topic,
+ final Collection<KeyValue<String, Long>> records) {
+ IntegrationTestUtils.produceKeyValuesSynchronously(
+ topic,
+ records,
+ producerConfig,
+ CLUSTER.time
+ );
+ }
+
+
+ @Test
+ public void shouldPauseAndResumeKafkaStreams() throws Exception {
+ // Create KafkaStreams instance
+ final StreamsBuilder builder = new StreamsBuilder();
+ builder.stream(INPUT_STREAM_1).groupByKey().count().toStream().to(OUTPUT_STREAM_1);
+
+ kafkaStreams = new KafkaStreams(builder.build(props()), props());
+ kafkaStreams.start();
+ waitForApplicationState(singletonList(kafkaStreams), State.RUNNING, STARTUP_TIMEOUT);
+
+ // Write data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify output
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 3),
+ CoreMatchers.equalTo(COUNT_OUTPUT_DATA));
+
+ // Pause KafkaStreams/topology
+ kafkaStreams.pause();
+ assertTrue(kafkaStreams.isPaused());
+
+ // Write more data
+ produceToInputTopics(INPUT_STREAM_1, STANDARD_INPUT_DATA);
+
+ // Verify that consumers read new data -- AKA, there is no lag.
+ final Map<String, Map<Integer, LagInfo>> lagMap = kafkaStreams.allLocalStorePartitionLags();
+ assertNoLag(lagMap);
+
+ // Verify no output somehow?
+ // Is there a better way to show this?
+ assertThat(waitUntilMinKeyValueRecordsReceived(consumerConfig, OUTPUT_STREAM_1, 0),
+ CoreMatchers.equalTo(Collections.emptyList()));
Review Comment:
As you write in the comments, this is not straightforward. I think it is not good to just wait for no outputs for two reasons:
- it increases the duration of the test unnecessarily
- it opens the door for flakiness because we only rely on a duration for verification
I have the following idea. You create two identical `KafkaStreams` clients (be careful to specify distinct state directories for each). Both clients read from the same input topics but write to distinct output topics. You do the initial setup verifications for both. Then you pause one of the clients and wait until the other client produced a given amount of records to the output topic. When that happened, you verify if the paused client has not written anything to its output topic. The assumption is that both clients produce with the same rate to their output topics if they are not paused. If one of the two is paused and does not produce any records to the output topic in the same time the other client produces a certain number of records we can assume the pausing works.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/StreamThreadTest.java:
##########
@@ -1761,6 +1761,96 @@ public void shouldUpdateStandbyTask() throws Exception {
thread.taskManager().shutdown(true);
}
+ @SuppressWarnings("unchecked")
+ @Test
+ public void shouldNotUpdateStandbyTaskWhenPaused() throws Exception {
+ final String storeName1 = "count-one";
+ final String storeName2 = "table-two";
+ final String changelogName1 = APPLICATION_ID + "-" + storeName1 + "-changelog";
+ final String changelogName2 = APPLICATION_ID + "-" + storeName2 + "-changelog";
+ final TopicPartition partition1 = new TopicPartition(changelogName1, 1);
+ final TopicPartition partition2 = new TopicPartition(changelogName2, 1);
+ internalStreamsBuilder
+ .stream(Collections.singleton(topic1), consumed)
+ .groupByKey()
+ .count(Materialized.as(storeName1));
+ final MaterializedInternal<Object, Object, KeyValueStore<Bytes, byte[]>> materialized
+ = new MaterializedInternal<>(Materialized.as(storeName2), internalStreamsBuilder, "");
+ internalStreamsBuilder.table(topic2, new ConsumedInternal<>(), materialized);
+
+ internalStreamsBuilder.buildAndOptimizeTopology();
+ final StreamThread thread = createStreamThread(CLIENT_ID, config, false);
+ final MockConsumer<byte[], byte[]> restoreConsumer = clientSupplier.restoreConsumer;
+ restoreConsumer.updatePartitions(changelogName1,
+ Collections.singletonList(new PartitionInfo(changelogName1, 1, null, new Node[0], new Node[0]))
+ );
+
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition1, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition1, 0L));
+ restoreConsumer.updateEndOffsets(Collections.singletonMap(partition2, 10L));
+ restoreConsumer.updateBeginningOffsets(Collections.singletonMap(partition2, 0L));
+ final OffsetCheckpoint checkpoint
+ = new OffsetCheckpoint(new File(stateDirectory.getOrCreateDirectoryForTask(task3), CHECKPOINT_FILE_NAME));
+ checkpoint.write(Collections.singletonMap(partition2, 5L));
+
+ thread.setState(StreamThread.State.STARTING);
+ thread.rebalanceListener().onPartitionsRevoked(Collections.emptySet());
+
+ final Map<TaskId, Set<TopicPartition>> standbyTasks = new HashMap<>();
+
+ // assign single partition
+ standbyTasks.put(task1, Collections.singleton(t1p1));
+ standbyTasks.put(task3, Collections.singleton(t2p1));
+
+ thread.taskManager().handleAssignment(emptyMap(), standbyTasks);
+ thread.taskManager().tryToCompleteRestoration(mockTime.milliseconds(), null);
+
+ thread.rebalanceListener().onPartitionsAssigned(Collections.emptyList());
+
+ thread.runOnce();
+
+ final StandbyTask standbyTask1 = standbyTask(thread.taskManager(), t1p1);
+ final StandbyTask standbyTask2 = standbyTask(thread.taskManager(), t2p1);
+ assertEquals(task1, standbyTask1.id());
+ assertEquals(task3, standbyTask2.id());
+
+ final KeyValueStore<Object, Long> store1 = (KeyValueStore<Object, Long>) standbyTask1.getStore(storeName1);
+ final KeyValueStore<Object, Long> store2 = (KeyValueStore<Object, Long>) standbyTask2.getStore(storeName2);
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // let the store1 be restored from 0 to 10; store2 be restored from 5 (checkpointed) to 10
+ for (long i = 0L; i < 10L; i++) {
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName1,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ restoreConsumer.addRecord(new ConsumerRecord<>(
+ changelogName2,
+ 1,
+ i,
+ ("K" + i).getBytes(),
+ ("V" + i).getBytes()));
+ }
+
+ // Simulate pause
+ thread.taskManager().topologyMetadata().pauseTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+ assertEquals(0L, store1.approximateNumEntries());
+ assertEquals(0L, store2.approximateNumEntries());
+
+ // Simulate resume
+ thread.taskManager().topologyMetadata().resumeTopology(TopologyMetadata.UNNAMED_TOPOLOGY);
+ thread.runOnce();
+
+ assertEquals(10L, store1.approximateNumEntries());
+ assertEquals(4L, store2.approximateNumEntries());
Review Comment:
Ah, I see! Could you please extract common code (especially the setup code) of those two tests into a method.
We could consider to just check for greater 0.
Additionally to standbys, we should also test if active tasks in restoration are paused.
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskExecutionMetadataTest.java:
##########
@@ -0,0 +1,123 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.processor.internals;
+
+import static org.apache.kafka.streams.processor.internals.TopologyMetadata.UNNAMED_TOPOLOGY;
+import static org.mockito.Mockito.mock;
+import static org.mockito.Mockito.when;
+
+import java.util.Arrays;
+import java.util.Collections;
+import java.util.HashSet;
+import java.util.Set;
+import java.util.concurrent.ConcurrentHashMap;
+import org.apache.kafka.streams.processor.TaskId;
+import org.junit.Assert;
+import org.junit.Test;
+
+public class TaskExecutionMetadataTest {
+ final static String TOPOLOGY1 = "topology1";
+ final static String TOPOLOGY2 = "topology2";
+ final static Set<String> NAMED_TOPOLOGIES = new HashSet<>(Arrays.asList(TOPOLOGY1, TOPOLOGY2));
+
+ @Test
+ public void testCanProcessWithoutNamedTopologies() {
+ final Set<String> topologies = Collections.singleton(UNNAMED_TOPOLOGY);
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(topologies,
+ pausedTopologies);
+
+ final Task mockTask = createMockTask(UNNAMED_TOPOLOGY);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask, 0));
+ // This pauses an UNNAMED_TOPOLOGY / a KafkaStreams instance without named/modular
+ // topologies.
+ pausedTopologies.add(UNNAMED_TOPOLOGY);
+ Assert.assertFalse(metadata.canProcessTask(mockTask, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBePausedIndependently() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.add(TOPOLOGY1);
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBeStartedPaused() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+ pausedTopologies.add(TOPOLOGY1);
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ pausedTopologies.remove(TOPOLOGY1);
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+ }
+
+ @Test
+ public void testNamedTopologiesCanBackoff() {
+ final Set<String> pausedTopologies = ConcurrentHashMap.newKeySet();
+
+ final TaskExecutionMetadata metadata = new TaskExecutionMetadata(NAMED_TOPOLOGIES,
+ pausedTopologies);
+
+ final Task mockTask1 = createMockTask(TOPOLOGY1);
+ final Task mockTask2 = createMockTask(TOPOLOGY2);
+
+ Assert.assertTrue(metadata.canProcessTask(mockTask1, 0));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 0));
+
+ metadata.registerTaskError(mockTask1, new Throwable("Error"), 0);
+ // One second after the error, task1 cannot process, task2 can.
+ Assert.assertFalse(metadata.canProcessTask(mockTask1, 1000));
+ Assert.assertTrue(metadata.canProcessTask(mockTask2, 1000));
+
Review Comment:
Could you please use `CONSTANT_BACKOFF_MS - 1` for the verifications on line 111 and 112? I think it makes the intent clearer.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r887185726
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StoreChangelogReader.java:
##########
@@ -479,6 +485,26 @@ public void restore(final Map<TaskId, Task> tasks) {
}
}
+ private void updateStandbyPartitions(final Map<TaskId, Task> tasks,
Review Comment:
I'm wondering if we can make this a more general rule, like:
1) When reader is in Restore_Active state i.e. there are at least one (for sake of simplicity just say we happen to have exactly one) active task which needs restoration, say taskA; then taskA was paused, we should be able to transit to Update_Standby.
2) When reader is in Update_Standby state, and there is one active task say taskA resumed; we should be able to transit to Restore_Active.
I know for now it does not matter since we always pause all tasks with the current APIs, but this is extensible for finer-grained controls in the future.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879798075
##########
streams/src/main/java/org/apache/kafka/streams/KafkaStreams.java:
##########
@@ -1662,6 +1663,51 @@ public <T> T store(final StoreQueryParameters<T> storeQueryParameters) {
return queryableStoreProvider.getStore(storeQueryParameters);
}
+ /**
+ * This method pauses processing for the KafkaStreams instance.
+ *
+ * Paused topologies will only skip over a) processing, b) punctuation, and c) standby tasks.
+ * Notably, paused topologies will still poll Kafka consumers, and commit offsets.
+ * This method sets transient state that is not maintained or managed among instances.
+ * Note that pause() can be called before start() in order to start a KafkaStreams instance
+ * in a manner where the processing is paused as described, but the consumers are started up.
+ */
+ public void pause() {
Review Comment:
Yes! I added some in this commit https://github.com/apache/kafka/pull/12161/commits/e0a80bbf31b79a1eb81bd67dff33ce81933b3e73.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] jnh5y commented on a diff in pull request #12161: KAFKA-13873 Add ability to Pause / Resume KafkaStreams Topologies
Posted by GitBox <gi...@apache.org>.
jnh5y commented on code in PR #12161:
URL: https://github.com/apache/kafka/pull/12161#discussion_r879881585
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/StreamThread.java:
##########
@@ -897,7 +897,8 @@ private void initializeAndRestorePhase() {
}
// we can always let changelog reader try restoring in order to initialize the changelogs;
// if there's no active restoring or standby updating it would not try to fetch any data
- changelogReader.restore(taskManager.tasks());
+ // After KAFKA-13873, we only restore the not paused tasks.
+ changelogReader.restore(taskManager.notPausedTasks());
Review Comment:
I think I've covered this with a new test in StreamThreadTest.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org