You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org> on 2026/01/27 19:52:09 UTC
[PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Nikita-Shupletsov opened a new pull request, #21365:
URL: https://github.com/apache/kafka/pull/21365
Added logic to close pending tasks to init.
Made standby task closure similar to the one for active tasks. Added a separate method for getting standby tasks from task registry. Added an integration test that reproduces the issue.
Delete this text and replace it with a detailed description of your change. The
PR title and body will become the squashed commit message.
If you would like to tag individuals, add some commentary, upload images, or
include other supplemental information that should not be part of the eventual
commit message, please use a separate comment.
If applicable, please include a summary of the testing strategy (including
rationale) for the proposed change. Unit and/or integration tests are expected
for any behavior change and system tests should be considered for larger
changes.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3863122675
4.0: https://github.com/apache/kafka/pull/21421
4.1: https://github.com/apache/kafka/pull/21422
4.2: https://github.com/apache/kafka/pull/21423
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744816224
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
Review Comment:
I was talking about this code: https://github.com/apache/kafka/blob/trunk/clients/src/main/java/org/apache/kafka/clients/consumer/internals/ClassicKafkaConsumer.java#L639-L671
so updateAssignmentMetadataIfNeeded calls onAssignment and so on, which in turn at some point will call clearCache. where we set up a synchronization point in this test.
however if pollForFetches returned nothing, we will retry the whole thing again, which will call onAssignment again, and it will cause problems in the test, because we will not know how many times we get this onAssigment called. it can be one, it can two, it can be more. so in order to always call it once, we want pollForFetches to return something to break the loop right after we get to the right place(in our case it's clearCache)
I will update the comment to make it more clear what I mean
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2759245575
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
Review Comment:
See other comment.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -193,7 +193,9 @@ public synchronized void removeTask(final Task taskToRemove) {
throw new IllegalStateException("Attempted to remove a task that is not closed or suspended: " + taskId);
}
- if (taskToRemove.isActive()) {
+ if (pendingTasksToInit.contains(taskToRemove)) {
Review Comment:
The standby tasks should indeed by in CREATED in that collection. Sounds like a useful invariant to have, maybe we can even add a check in `addPendingTasksToInit` ?
Using drainPendingTasksToInit sounds like a good idea to me. Not sure if there are any complications though, but it would seem cleaner code wise.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2771512180
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -3667,6 +3663,35 @@ public void shouldShutDownStateUpdaterAndCloseDirtyTasksFailedDuringRemoval() {
verify(removedFailedStandbyTaskDuringRemoval).closeDirty();
}
+ @Test
+ public void shouldShutDownPendingTasksToInit() {
+ final TasksRegistry tasks = mock(TasksRegistry.class);
+ final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks);
+
+ final StandbyTask standbyTask00 = standbyTask(taskId00, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
+ .withInputPartitions(taskId00Partitions)
+ .build();
+
+ final StreamTask activeTask01 = statefulTask(taskId01, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
+ .withInputPartitions(taskId00Partitions).build();
+
+ when(tasks.drainPendingStandbyTasksToInit()).thenReturn(Set.of(standbyTask00));
+ when(tasks.drainPendingActiveTasksToInit()).thenReturn(Set.of(activeTask01));
+
+ taskManager.shutdown(true);
+
+ verify(standbyTask00).prepareCommit(true);
+ verify(standbyTask00).postCommit(true);
+ verify(standbyTask00).suspend();
+ verify(standbyTask00).closeClean();
+
+ verify(activeTask01).prepareCommit(true);
+ verify(activeTask01).suspend();
Review Comment:
fixed
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3843718537
>But I wonder if we should rename "allTasks" to "allInitializedTasks" then?
@lucasbru would you mind if I did that in a followup PR?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2757015026
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KafkaStreamsWrapper.java:
##########
@@ -48,7 +48,11 @@ public List<StreamThread> streamThreads() {
public void setStreamThreadStateListener(final StreamThread.StateListener listener) {
if (state == State.CREATED) {
for (final StreamThread thread : threads) {
- thread.setStateListener(listener);
+ StreamThread.StateListener originalListener = thread.getStateListener();
+ thread.setStateListener((t, newState, oldState) -> {
+ originalListener.onChange(t, newState, oldState);
+ listener.onChange(t, newState, oldState);
+ });
Review Comment:
Oh. This is the thread-state listener, not the KS one... Missed this.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3841591722
> An alternative way to fix it would be, I think, to consider `pendingTasksToInit` as a subset of `Tasks.allTasks`. Then conceptually, it may be simpler to say "close all tasks". But we'd have to check carefully what other places `Tasks.allTasks` and related methods are used, and if we are changing one of those places. In that case, we'd still need the "remove" fix you did.
On second thought, that would probably create more problems than it solves. But I wonder if we should rename "allTasks" to "allInitializedTasks" then?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax merged PR #21365:
URL: https://github.com/apache/kafka/pull/21365
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3880604355
@lucasbru, @mjsax the follow-up PR to rename the methods in TaskRegistry: https://github.com/apache/kafka/pull/21448
thanks in advance!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2765781911
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1414,10 +1414,19 @@ void shutdown(final boolean clean) {
// TODO: change type to `StreamTask`
final Set<Task> activeTasks = new TreeSet<>(Comparator.comparing(Task::id));
activeTasks.addAll(tasks.activeTasks());
+ final Set<Task> standbyTasks = new TreeSet<>(Comparator.comparing(Task::id));
Review Comment:
nit: should this be `StrandbyTask` type (compare comment above about `StreamTask` type -- we would also just update the comment, and try to do a follow up PR to improve types later?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1414,10 +1414,19 @@ void shutdown(final boolean clean) {
// TODO: change type to `StreamTask`
final Set<Task> activeTasks = new TreeSet<>(Comparator.comparing(Task::id));
activeTasks.addAll(tasks.activeTasks());
+ final Set<Task> standbyTasks = new TreeSet<>(Comparator.comparing(Task::id));
+ standbyTasks.addAll(tasks.standbyTasks());
+
+ Set<Task> pendingActiveTasks = tasks.drainPendingActiveTasksToInit();
Review Comment:
Seems this should be `final` -- wondering why the build did not fail with a checktsyle error? Can we check this?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
Review Comment:
Not sure why this test does not verify the locking for `taskId01` ?
Or is the test setup with two task just not necessary and using one task is sufficient?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -3667,6 +3663,35 @@ public void shouldShutDownStateUpdaterAndCloseDirtyTasksFailedDuringRemoval() {
verify(removedFailedStandbyTaskDuringRemoval).closeDirty();
}
+ @Test
+ public void shouldShutDownPendingTasksToInit() {
+ final TasksRegistry tasks = mock(TasksRegistry.class);
+ final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks);
+
+ final StandbyTask standbyTask00 = standbyTask(taskId00, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
+ .withInputPartitions(taskId00Partitions)
+ .build();
+
+ final StreamTask activeTask01 = statefulTask(taskId01, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
Review Comment:
as above
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,236 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} return false
Review Comment:
```suggestion
* <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
```
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TasksTest.java:
##########
@@ -230,4 +249,18 @@ public void shouldClearAllPendingTasks() {
assertTrue(tasks.pendingActiveTasksToCreate().isEmpty());
assertTrue(tasks.pendingStandbyTasksToCreate().isEmpty());
}
+
+ @Test
+ public void shouldRemovePendingTaskToClose() {
+ final StreamTask activeTask1 = statefulTask(TASK_0_0, Set.of(TOPIC_PARTITION_B_0))
+ .inState(State.SUSPENDED).build();
+ tasks.addPendingTasksToClose(List.of(activeTask1));
+
+ tasks.removeTask(activeTask1);
+ assertFalse(tasks.pendingTasksToInit().contains(activeTask1));
+ assertFalse(tasks.allTasks().contains(activeTask1));
+
+ tasks.addPendingTasksToClose(List.of(activeTask1));
+ assertTrue(tasks.pendingTasksToClose().contains(activeTask1));
Review Comment:
Why do we add it back and verify? Could we not do the verification above, right after the added the task the first time?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -312,12 +312,8 @@ public void shouldLockTasksOnClose() {
final StreamTask activeTask1 = statefulTask(taskId00, taskId00ChangelogPartitions)
.inState(State.RUNNING)
.withInputPartitions(taskId00Partitions).build();
- final StreamTask activeTask2 = statefulTask(taskId01, taskId01ChangelogPartitions)
- .inState(State.RUNNING)
- .withInputPartitions(taskId01Partitions).build();
final TasksRegistry tasks = mock(TasksRegistry.class);
final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks, true);
- when(tasks.allTasks()).thenReturn(Set.of(activeTask1, activeTask2));
Review Comment:
We remove this, because `allTasks()` is not called any longer?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1414,10 +1414,19 @@ void shutdown(final boolean clean) {
// TODO: change type to `StreamTask`
final Set<Task> activeTasks = new TreeSet<>(Comparator.comparing(Task::id));
activeTasks.addAll(tasks.activeTasks());
+ final Set<Task> standbyTasks = new TreeSet<>(Comparator.comparing(Task::id));
+ standbyTasks.addAll(tasks.standbyTasks());
+
+ Set<Task> pendingActiveTasks = tasks.drainPendingActiveTasksToInit();
+ activeTasks.addAll(pendingActiveTasks);
+ tasks.addPendingTasksToClose(pendingActiveTasks);
+ Set<Task> pendingStandbyTasks = tasks.drainPendingStandbyTasksToInit();
Review Comment:
Same. Should be `final`
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -3667,6 +3663,35 @@ public void shouldShutDownStateUpdaterAndCloseDirtyTasksFailedDuringRemoval() {
verify(removedFailedStandbyTaskDuringRemoval).closeDirty();
}
+ @Test
+ public void shouldShutDownPendingTasksToInit() {
+ final TasksRegistry tasks = mock(TasksRegistry.class);
+ final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks);
+
+ final StandbyTask standbyTask00 = standbyTask(taskId00, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
+ .withInputPartitions(taskId00Partitions)
+ .build();
+
+ final StreamTask activeTask01 = statefulTask(taskId01, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
+ .withInputPartitions(taskId00Partitions).build();
+
+ when(tasks.drainPendingStandbyTasksToInit()).thenReturn(Set.of(standbyTask00));
+ when(tasks.drainPendingActiveTasksToInit()).thenReturn(Set.of(activeTask01));
+
+ taskManager.shutdown(true);
+
+ verify(standbyTask00).prepareCommit(true);
+ verify(standbyTask00).postCommit(true);
+ verify(standbyTask00).suspend();
+ verify(standbyTask00).closeClean();
+
+ verify(activeTask01).prepareCommit(true);
+ verify(activeTask01).suspend();
Review Comment:
Why is there no `postCommit()` for the active task?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -139,6 +154,21 @@ public boolean hasPendingTasksToInit() {
return !pendingTasksToInit.isEmpty();
}
+ @Override
+ public Set<Task> pendingTasksToClose() {
+ return Collections.unmodifiableSet(pendingTasksToClose);
+ }
+
+ @Override
+ public void addPendingTasksToClose(Collection<Task> tasks) {
Review Comment:
Parameter should be `final`
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -3667,6 +3663,35 @@ public void shouldShutDownStateUpdaterAndCloseDirtyTasksFailedDuringRemoval() {
verify(removedFailedStandbyTaskDuringRemoval).closeDirty();
}
+ @Test
+ public void shouldShutDownPendingTasksToInit() {
+ final TasksRegistry tasks = mock(TasksRegistry.class);
+ final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks);
+
+ final StandbyTask standbyTask00 = standbyTask(taskId00, taskId00ChangelogPartitions)
+ .inState(State.RUNNING)
Review Comment:
Should the state be `CREATED`?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -3667,6 +3663,35 @@ public void shouldShutDownStateUpdaterAndCloseDirtyTasksFailedDuringRemoval() {
verify(removedFailedStandbyTaskDuringRemoval).closeDirty();
}
+ @Test
+ public void shouldShutDownPendingTasksToInit() {
Review Comment:
```suggestion
public void shouldClosePendingTasksToInitDuringShutdown() {
```
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -312,12 +312,8 @@ public void shouldLockTasksOnClose() {
final StreamTask activeTask1 = statefulTask(taskId00, taskId00ChangelogPartitions)
.inState(State.RUNNING)
.withInputPartitions(taskId00Partitions).build();
- final StreamTask activeTask2 = statefulTask(taskId01, taskId01ChangelogPartitions)
Review Comment:
Why are we removing this task from the test? Is using a single task sufficient (and it's unclear why we did use two tasks to begin with)?
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TasksTest.java:
##########
@@ -164,6 +166,23 @@ public void shouldVerifyIfPendingActiveTaskToInitAreDrained() {
assertTrue(tasks.pendingTasksToInit().containsAll(Set.of(standbyTask1, standbyTask2)));
}
+ @Test
+ public void shouldVerifyIfPendingStandbyTaskToInitAreDrained() {
+ final StreamTask activeTask1 = statefulTask(TASK_0_0, Set.of(TOPIC_PARTITION_B_0)).build();
+ final StreamTask activeTask2 = statefulTask(TASK_0_1, Set.of(TOPIC_PARTITION_B_1)).build();
+ final StandbyTask standbyTask1 = standbyTask(TASK_1_0, Set.of(TOPIC_PARTITION_A_0)).build();
+ final StandbyTask standbyTask2 = standbyTask(TASK_1_1, Set.of(TOPIC_PARTITION_A_1)).build();
+ tasks.addPendingTasksToInit(Set.of(activeTask1, activeTask2, standbyTask1, standbyTask2));
+
+ final Set<Task> standbyTasksToInit = tasks.drainPendingStandbyTasksToInit();
+ assertEquals(2, standbyTasksToInit.size());
Review Comment:
```suggestion
assertEquals(2, standbyTasksToInit.size());
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744349397
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
Review Comment:
This is the default IIRC -- no need to set it
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
Review Comment:
```suggestion
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
Review Comment:
Not sure if I understand. "disable retires on the consumer" -- what retires are you referring to, and what consumer exaclty?
"it retires the whole sequence of actions, including the rebalance data" -- what sequence are you referring too? What do you mean by "rebalance data"?
"which we don't want, because we just staged the right condition" -- Why do we not want this? Not sure what the "right condition" is, and why we would "lose it" w/o sending an input record.
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
Review Comment:
Why not `return super.setFlushListener(...)`?
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
Review Comment:
Might be good to add a test description at the top, explaining what we are exactly testing.
```
Starting two KS app
Blocking the first KS app during the rebalance (which is started when the second apps starts) because <explanation>
```
and so forth. I am not sure I understand the test yet, and I am sure if somebody come back to it at some point in the future they might struggle, too.
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
Review Comment:
Same. Already the default.
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
Review Comment:
"before it gets to moving messages from task registry to state updater" -- do you mean "task" instead of "messaged"?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -193,7 +193,9 @@ public synchronized void removeTask(final Task taskToRemove) {
throw new IllegalStateException("Attempted to remove a task that is not closed or suspended: " + taskId);
}
- if (taskToRemove.isActive()) {
+ if (pendingTasksToInit.contains(taskToRemove)) {
Review Comment:
We throw above if a task is not CLOSED or SUSPENDED, but `pendingTaskToInit` should be in state `CREATED` ? -- Do we need to update the above condition?
Are we testing this code path properly? If no test fails, it seems we never call `removeTask()` with a pending task?
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
+ }
+
+ @Override
+ public void flushCache() {
+ }
Review Comment:
Should we call `super.flushCache()`?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1523,7 +1532,7 @@ private Collection<Task> tryCloseCleanActiveTasks(final Collection<Task> activeT
final boolean clean,
final AtomicReference<RuntimeException> firstException) {
if (!clean) {
- return activeTaskIterable();
+ return activeTasksToClose;
Review Comment:
For my own understanding. The original code here was not wrong in the sense that before this PR the `activeTasksToClose` that got passed in was the same as `activeTaskIterable()`? But with this PR, we also include pending task in `activeTasksToClose`, and that's why we need to update the code here?
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
+ }
+
+ @Override
+ public void flushCache() {
+ }
+
+ @Override
+ public void clearCache() {
Review Comment:
Should we call `super.clearCache()` at some point (not sure if in the beginning or at the end of the method)?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
Review Comment:
Can you explain why we also need a fix for this method in more details? It make sense to me, that we need to fix `shutdown()`, but not totally sure about this one.
Not saying we don't need to fix it, I just don't see the full picture of the bug yet.
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KafkaStreamsWrapper.java:
##########
@@ -48,7 +48,11 @@ public List<StreamThread> streamThreads() {
public void setStreamThreadStateListener(final StreamThread.StateListener listener) {
if (state == State.CREATED) {
for (final StreamThread thread : threads) {
- thread.setStateListener(listener);
+ StreamThread.StateListener originalListener = thread.getStateListener();
+ thread.setStateListener((t, newState, oldState) -> {
+ originalListener.onChange(t, newState, oldState);
+ listener.onChange(t, newState, oldState);
+ });
Review Comment:
Why do we need this change? It seems you try to allow to register a second state listener? In general, we allow to only register one -- why do we need two? (Also, if we need multiple, why limit to two?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3843464262
> maybe we would need to add a new member to TasksRegistry taking "pending tasks" which got closed?
Yes, that is what I mean by this -- after we drain the pending-tasks-to-init, we would add them into some new "pending-task-to-close" member inside `TaskRegistry` and update `removeTask` so check ownership based on this new member.
It's to some extend "cosmetics" as we only move all "pending task" from one collection into a different one, but it preserve the invariant we have, and makes the code cleaner.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2771481367
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -312,12 +312,8 @@ public void shouldLockTasksOnClose() {
final StreamTask activeTask1 = statefulTask(taskId00, taskId00ChangelogPartitions)
.inState(State.RUNNING)
.withInputPartitions(taskId00Partitions).build();
- final StreamTask activeTask2 = statefulTask(taskId01, taskId01ChangelogPartitions)
- .inState(State.RUNNING)
- .withInputPartitions(taskId01Partitions).build();
final TasksRegistry tasks = mock(TasksRegistry.class);
final TaskManager taskManager = setUpTaskManager(ProcessingMode.AT_LEAST_ONCE, tasks, true);
- when(tasks.allTasks()).thenReturn(Set.of(activeTask1, activeTask2));
Review Comment:
yes. previously during unclean shutdown we would call activeTaskIterable, which does `stateUpdater.tasks().stream().filter(Task::isActive)`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2771485439
##########
streams/src/test/java/org/apache/kafka/streams/processor/internals/TaskManagerTest.java:
##########
@@ -312,12 +312,8 @@ public void shouldLockTasksOnClose() {
final StreamTask activeTask1 = statefulTask(taskId00, taskId00ChangelogPartitions)
.inState(State.RUNNING)
.withInputPartitions(taskId00Partitions).build();
- final StreamTask activeTask2 = statefulTask(taskId01, taskId01ChangelogPartitions)
Review Comment:
see the comment bellow.
tbh, I am not 100% sure why it had two tasks, as we don't check the second task at all. it might have been copy pasted from another test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3863009888
Thanks for the fix. -- Merged to `trunk`.
Cherry-picking to older branches is not clean. Can you maybe look into it and open new PRs all the way back to `4.2., `4.1`, and `4.0` branches?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744771114
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -193,7 +193,9 @@ public synchronized void removeTask(final Task taskToRemove) {
throw new IllegalStateException("Attempted to remove a task that is not closed or suspended: " + taskId);
}
- if (taskToRemove.isActive()) {
+ if (pendingTasksToInit.contains(taskToRemove)) {
Review Comment:
it's supposed to be CLOSED too.
I added it because in TaskManager when we close a task, we remove it from the task registry(e.g. https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L1405), so we expect the pending task to be closed as well.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2757103115
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
@@ -193,7 +193,9 @@ public synchronized void removeTask(final Task taskToRemove) {
throw new IllegalStateException("Attempted to remove a task that is not closed or suspended: " + taskId);
}
- if (taskToRemove.isActive()) {
+ if (pendingTasksToInit.contains(taskToRemove)) {
Review Comment:
I see -- it's because we add all pending tasks for closing above:
```
for (Task pendingTask : tasks.pendingTasksToInit()) {
```
So these tasks are in CREATED when we get them, but we first CLOSE them and then remove them. I am just wondering if this violates the invariant that all tasks in `pendingTasksToInit` _should_ be in CREATED stated (not 100% if this is an actual contract we follow, but it would make sense IMHO), and if we should use `drainPendingTasksToInit` instead of `pendingTasksToInit`, because we intent to close these tasks -- this would avoid that we need to remove these tasks here, as `drainPendingTasksToInit` would do the removal for us already.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1414,10 +1414,19 @@ void shutdown(final boolean clean) {
// TODO: change type to `StreamTask`
final Set<Task> activeTasks = new TreeSet<>(Comparator.comparing(Task::id));
activeTasks.addAll(tasks.activeTasks());
+ final Set<Task> standbyTasks = new TreeSet<>(Comparator.comparing(Task::id));
+ standbyTasks.addAll(tasks.standbyTasks());
+ for (Task pendingTask : tasks.pendingTasksToInit()) {
+ if (pendingTask.isActive()) {
+ activeTasks.add(pendingTask);
+ } else {
+ standbyTasks.add(pendingTask);
+ }
+ }
Review Comment:
Thanks for updating the PR description. It say "shutdown during rebalance when active task become standby tasks" but seems it goes either way, and the PR is actually fixing both direction?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
Review Comment:
Cf my comment below (https://github.com/apache/kafka/pull/21365/changes#r2757103115) -- maybe we would need to add a new member to `TasksRegistry` taking "pending tasks" which got closed?
Would like to hear from @lucasbru about this, as he know this part of the code better. I am just worried to create some mess here (maybe I am wrong). -- Maybe it's also ok to just "reuse" `pendingTaskToInit` to also track these tasks _after_ they got CLOSED. Just not sure about it.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1523,7 +1532,7 @@ private Collection<Task> tryCloseCleanActiveTasks(final Collection<Task> activeT
final boolean clean,
final AtomicReference<RuntimeException> firstException) {
if (!clean) {
- return activeTaskIterable();
+ return activeTasksToClose;
Review Comment:
> even without my change it's already a bit incorrect
I see -- so it's not a problem currently, but only because of ordering (ie, "luck"), but if ordering changes, it would break. Thanks for clarifying.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744796758
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/Tasks.java:
##########
Review Comment:
as I mentioned above, this method is called during shutdown when we finish closing a task(cleanly or dirtily). without this change this method will be throwing `Attempted to remove an active task that is not owned` or `Attempted to remove a standby task that is not owned`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744794087
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1523,7 +1532,7 @@ private Collection<Task> tryCloseCleanActiveTasks(final Collection<Task> activeT
final boolean clean,
final AtomicReference<RuntimeException> firstException) {
if (!clean) {
- return activeTaskIterable();
+ return activeTasksToClose;
Review Comment:
the reason why I decided to change this was that if we follow the call:
* https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L1405, here we pass two lists of tasks to close. the active ones used to accept activeTaskIterable as well, but it was changed at some point.
* then we get into closeAndCleanUpTasks, then into tryCloseCleanActiveTasks, tryCloseCleanStandbyTasks
* at this point we check if we even need to try to close them cleanly. if not `Returns the set of active tasks that must be closed dirty`
* at this point it makes sense to return the list we received, not a different list. because it's a potential place for a bug: we ask the method to close one set of tasks, but it closes a different set of tasks.
even without my change it's already a bit incorrect, because we take the active tasks from the task registry and from the state updater: https://github.com/apache/kafka/blob/trunk/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L1708-L1709, which isn't really a big deal, because we handle the state updater before this call, so there should be nothing there.
but if it changes, it will be pretty hard to notice that slight change in the behavior.
on top of that, as you mentioned, this list doesn't contain the pending tasks. so if we keep it as is, when we shutdown dirtily, we will not close them.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744755854
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
+ }
+
+ @Override
+ public void flushCache() {
+ }
+
+ @Override
+ public void clearCache() {
Review Comment:
as we just implement CachedStateStore, there is no super
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744762670
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/KafkaStreamsWrapper.java:
##########
@@ -48,7 +48,11 @@ public List<StreamThread> streamThreads() {
public void setStreamThreadStateListener(final StreamThread.StateListener listener) {
if (state == State.CREATED) {
for (final StreamThread thread : threads) {
- thread.setStateListener(listener);
+ StreamThread.StateListener originalListener = thread.getStateListener();
+ thread.setStateListener((t, newState, oldState) -> {
+ originalListener.onChange(t, newState, oldState);
+ listener.onChange(t, newState, oldState);
+ });
Review Comment:
because it overrides the default one, and KafkaStreams#state doesn't work.
with this change if we add a special state listener, we keep the old one as well.
if for some reason we would like to add two additional state listeners, this implementation will work too. we will just need to call it twice. then it will wrap it twice and all three listeners will be called instead of one.
the reason I changed the method was pretty much because I wanted to have a listener and a to be able to call KafkaStreams#state and get the actual state as well
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "mjsax (via GitHub)" <gi...@apache.org>.
mjsax commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2772196526
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
+ * <li>shutdowns the first KS app and waits for the stream tread to get into PENDING_SHUTDOWN state</li>
+ * <li>releases the lock</li>
+ * </ul><p>
+ * At this point {@link org.apache.kafka.streams.processor.internals.TaskManager#shutdown} will be called,
+ * and we will have a pending task to init with an open store(because tasks keep their stores open during recycle).
+ * <p>
+ * This test verifies that the open store is closed during shutdown.
+ */
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message with wrong key and value types to trigger a stream thread failure and avoid the second rebalance
Review Comment:
```suggestion
// sending a message with wrong key and value types to trigger a stream thread failure and avoid the second rebalance
// note that writing this message does not trigger the crash right away -- the thread is still blocked inside `poll()` waiting for the shutdown latch to unlock to complete the previous, still ongoing rebalance
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
Review Comment:
```suggestion
* Second, we need to avoid the second rebalance, to avoid that the pending tasks is closed during such a rebalance, ie, before we enter the shutdown phase.
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
+ * <li>shutdowns the first KS app and waits for the stream tread to get into PENDING_SHUTDOWN state</li>
Review Comment:
```suggestion
* <li>shutdowns the first KS app</li>
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
Review Comment:
```suggestion
* <li>sends a message with wrong types to crash the stream thread (this avoids a second rebalance, and enters shutdown directly)</li>
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
+ * <li>shutdowns the first KS app and waits for the stream tread to get into PENDING_SHUTDOWN state</li>
+ * <li>releases the lock</li>
+ * </ul><p>
+ * At this point {@link org.apache.kafka.streams.processor.internals.TaskManager#shutdown} will be called,
+ * and we will have a pending task to init with an open store(because tasks keep their stores open during recycle).
+ * <p>
+ * This test verifies that the open store is closed during shutdown.
+ */
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message with wrong key and value types to trigger a stream thread failure and avoid the second rebalance
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>("key", 1L)),
+ TestUtils.producerConfig(cluster.bootstrapServers(), StringSerializer.class, LongSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving tasks from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
+ }
+
+ @Override
+ public void flushCache() {
+ }
+
+ @Override
+ public void clearCache() {
+ // Clear cache is called during recycle, so we use it as a hook
+ recycleLatch.countDown();
+ try {
+ pendingShutdownLatch.await();
Review Comment:
```suggestion
// after we signaled via recycleLatch, that the task was converted into a "pending task",
// we block the rebalance to complete, until we get the shutdown signal,
// to avoid that the "pending task" get fully initialized
// (otherwise, we don't have a pending task when the shutdown happens)
pendingShutdownLatch.await();
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
+ * <li>shutdowns the first KS app and waits for the stream tread to get into PENDING_SHUTDOWN state</li>
+ * <li>releases the lock</li>
+ * </ul><p>
+ * At this point {@link org.apache.kafka.streams.processor.internals.TaskManager#shutdown} will be called,
+ * and we will have a pending task to init with an open store(because tasks keep their stores open during recycle).
+ * <p>
+ * This test verifies that the open store is closed during shutdown.
+ */
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message with wrong key and value types to trigger a stream thread failure and avoid the second rebalance
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>("key", 1L)),
+ TestUtils.producerConfig(cluster.bootstrapServers(), StringSerializer.class, LongSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
Review Comment:
```suggestion
// Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
// After the rebalance finished, the "poison pill" record gets picked up crashing the thread,
// and starting the shutdown directly
```
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,237 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ /**
+ * The conditions that we need to meet:
+ * <p><ul>
+ * <li>There is a task with an open store in {@link org.apache.kafka.streams.processor.internals.TasksRegistry#pendingTasksToInit}</li>
+ * <li>StreamThread gets into PENDING_SHUTDOWN state, so that {@link StreamThread#isStartingRunningOrPartitionAssigned} returns false
+ * before we call {@link StreamThread#checkStateUpdater} that would move the task to the StateUpdater </li>
+ * </ul><p>
+ * If all conditions are met, {@code TaskManager} needs to correctly close the open store during shutdown.
+ * <p>
+ * In order to have a task with an open store in the pending task list we first need to have an active task that gets converted
+ * to a standby one during rebalance(see {@link org.apache.kafka.streams.processor.internals.TaskManager#closeAndRecycleTasks}).
+ * Second, we need to avoid the second rebalance.
+ * <p>
+ * For that this test:
+ * <p><ul>
+ * <li>starts a KS app and waits for it to fully start</li>
+ * <li>starts another KS app which will trigger reassignment</li>
+ * <li>waits for {@link CachedStateStore#clearCache} to be called(it's called during task recycle) and locks on it</li>
+ * <li>sends a message with wrong types to crash the stream thread</li>
+ * <li>shutdowns the first KS app and waits for the stream tread to get into PENDING_SHUTDOWN state</li>
+ * <li>releases the lock</li>
+ * </ul><p>
+ * At this point {@link org.apache.kafka.streams.processor.internals.TaskManager#shutdown} will be called,
+ * and we will have a pending task to init with an open store(because tasks keep their stores open during recycle).
+ * <p>
+ * This test verifies that the open store is closed during shutdown.
+ */
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message with wrong key and value types to trigger a stream thread failure and avoid the second rebalance
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>("key", 1L)),
+ TestUtils.producerConfig(cluster.bootstrapServers(), StringSerializer.class, LongSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving tasks from task registry to state updater.
Review Comment:
```suggestion
// We don't want to let the rebalance finish before we trigger the shutdown, because we want the stream thread to stop before it gets to moving pending tasks from task registry to state updater.
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2771493573
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1414,10 +1414,19 @@ void shutdown(final boolean clean) {
// TODO: change type to `StreamTask`
final Set<Task> activeTasks = new TreeSet<>(Comparator.comparing(Task::id));
activeTasks.addAll(tasks.activeTasks());
+ final Set<Task> standbyTasks = new TreeSet<>(Comparator.comparing(Task::id));
Review Comment:
left a todo
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on code in PR #21365:
URL: https://github.com/apache/kafka/pull/21365#discussion_r2744755397
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
Review Comment:
as we just implement CachedStateStore, there is no super
##########
streams/integration-tests/src/test/java/org/apache/kafka/streams/integration/RebalanceTaskClosureIntegrationTest.java:
##########
@@ -0,0 +1,213 @@
+/*
+ * Licensed to the Apache Software Foundation (ASF) under one or more
+ * contributor license agreements. See the NOTICE file distributed with
+ * this work for additional information regarding copyright ownership.
+ * The ASF licenses this file to You under the Apache License, Version 2.0
+ * (the "License"); you may not use this file except in compliance with
+ * the License. You may obtain a copy of the License at
+ *
+ * http://www.apache.org/licenses/LICENSE-2.0
+ *
+ * Unless required by applicable law or agreed to in writing, software
+ * distributed under the License is distributed on an "AS IS" BASIS,
+ * WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+ * See the License for the specific language governing permissions and
+ * limitations under the License.
+ */
+package org.apache.kafka.streams.integration;
+
+import org.apache.kafka.clients.consumer.ConsumerConfig;
+import org.apache.kafka.common.serialization.LongSerializer;
+import org.apache.kafka.common.serialization.Serdes;
+import org.apache.kafka.common.serialization.StringSerializer;
+import org.apache.kafka.common.utils.Bytes;
+import org.apache.kafka.common.utils.MockTime;
+import org.apache.kafka.streams.CloseOptions;
+import org.apache.kafka.streams.KafkaStreams;
+import org.apache.kafka.streams.KeyValue;
+import org.apache.kafka.streams.StreamsConfig;
+import org.apache.kafka.streams.TopologyWrapper;
+import org.apache.kafka.streams.integration.utils.EmbeddedKafkaCluster;
+import org.apache.kafka.streams.integration.utils.IntegrationTestUtils;
+import org.apache.kafka.streams.processor.StateStore;
+import org.apache.kafka.streams.processor.StateStoreContext;
+import org.apache.kafka.streams.processor.internals.StreamThread;
+import org.apache.kafka.streams.state.KeyValueStore;
+import org.apache.kafka.streams.state.StoreBuilder;
+import org.apache.kafka.streams.state.internals.AbstractStoreBuilder;
+import org.apache.kafka.streams.state.internals.CacheFlushListener;
+import org.apache.kafka.streams.state.internals.CachedStateStore;
+import org.apache.kafka.streams.state.internals.RocksDBStore;
+import org.apache.kafka.test.MockApiProcessorSupplier;
+import org.apache.kafka.test.TestUtils;
+import org.junit.jupiter.api.AfterEach;
+import org.junit.jupiter.api.BeforeEach;
+import org.junit.jupiter.api.Test;
+import org.junit.jupiter.api.TestInfo;
+
+import java.io.IOException;
+import java.time.Duration;
+import java.util.List;
+import java.util.Properties;
+import java.util.concurrent.CountDownLatch;
+import java.util.concurrent.atomic.AtomicInteger;
+
+import static org.apache.kafka.streams.utils.TestUtils.safeUniqueTestName;
+import static org.junit.jupiter.api.Assertions.assertEquals;
+
+public class RebalanceTaskClosureIntegrationTest {
+
+ private static final int NUM_BROKERS = 1;
+ protected static final String INPUT_TOPIC_NAME = "input-topic";
+ private static final int NUM_PARTITIONS = 3;
+
+ private final EmbeddedKafkaCluster cluster = new EmbeddedKafkaCluster(NUM_BROKERS);
+
+ private KafkaStreamsWrapper streams1;
+ private KafkaStreamsWrapper streams2;
+ private String safeTestName;
+
+ @BeforeEach
+ public void before(final TestInfo testInfo) throws InterruptedException, IOException {
+ cluster.start();
+ cluster.createTopic(INPUT_TOPIC_NAME, NUM_PARTITIONS, 1);
+ safeTestName = safeUniqueTestName(testInfo);
+
+ }
+
+ @AfterEach
+ public void after() {
+ cluster.stop();
+ if (streams1 != null) {
+ streams1.close(Duration.ofSeconds(30));
+ }
+ if (streams2 != null) {
+ streams2.close(Duration.ofSeconds(30));
+ }
+ }
+
+ @Test
+ public void shouldClosePendingTasksToInitAfterRebalance() throws Exception {
+ final CountDownLatch recycleLatch = new CountDownLatch(1);
+ final CountDownLatch pendingShutdownLatch = new CountDownLatch(1);
+ // Count how many times we initialize and close stores
+ final AtomicInteger initCount = new AtomicInteger();
+ final AtomicInteger closeCount = new AtomicInteger();
+ final StoreBuilder<KeyValueStore<Bytes, byte[]>> storeBuilder = new AbstractStoreBuilder<>("testStateStore", Serdes.Integer(), Serdes.ByteArray(), new MockTime()) {
+
+ @Override
+ public KeyValueStore<Bytes, byte[]> build() {
+ return new TestRocksDBStore(name, recycleLatch, pendingShutdownLatch, initCount, closeCount);
+ }
+ };
+
+ final TopologyWrapper topology = new TopologyWrapper();
+ topology.addSource("ingest", INPUT_TOPIC_NAME);
+ topology.addProcessor("my-processor", new MockApiProcessorSupplier<>(), "ingest");
+ topology.addStateStore(storeBuilder, "my-processor");
+
+ streams1 = new KafkaStreamsWrapper(topology, props("1"));
+ streams1.setStreamThreadStateListener((t, newState, oldState) -> {
+ if (newState == StreamThread.State.PENDING_SHUTDOWN) {
+ pendingShutdownLatch.countDown();
+ }
+ });
+ streams1.start();
+
+ TestUtils.waitForCondition(() -> streams1.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ streams2 = new KafkaStreamsWrapper(topology, props("2"));
+ streams2.start();
+
+ TestUtils.waitForCondition(() -> streams2.state() == KafkaStreams.State.RUNNING, "Streams never reached RUNNING state");
+
+ // starting the second KS app triggered a rebalance. Which in turn will recycle active tasks that need to become standby.
+ // That's exactly what we are waiting for
+ recycleLatch.await();
+
+ // sending a message to disable retries in the consumer client. if there are no messages, it retries the whole sequence of actions,
+ // including the rebalance data. which we don't want, because we just staged the right condition
+ IntegrationTestUtils.produceKeyValuesSynchronously(INPUT_TOPIC_NAME, List.of(new KeyValue<>(1L, "key")),
+ TestUtils.producerConfig(cluster.bootstrapServers(), LongSerializer.class, StringSerializer.class, new Properties()), cluster.time);
+ // Now we can close both apps. The StreamThreadStateListener will unblock the clearCache call, letting the rebalance finish.
+ // We don't want it to happen any sooner, because we want the stream thread to stop before it gets to moving messages from task registry to state updater.
+ streams1.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+ streams2.close(CloseOptions.groupMembershipOperation(CloseOptions.GroupMembershipOperation.LEAVE_GROUP));
+
+ assertEquals(initCount.get(), closeCount.get());
+ }
+
+ private Properties props(final String storePathSuffix) {
+ final Properties streamsConfiguration = new Properties();
+
+ streamsConfiguration.put(StreamsConfig.APPLICATION_ID_CONFIG, safeTestName);
+ streamsConfiguration.put(StreamsConfig.BOOTSTRAP_SERVERS_CONFIG, cluster.bootstrapServers());
+ streamsConfiguration.put(ConsumerConfig.AUTO_OFFSET_RESET_CONFIG, "earliest");
+ streamsConfiguration.put(ConsumerConfig.DEFAULT_API_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(ConsumerConfig.REQUEST_TIMEOUT_MS_CONFIG, 1000);
+ streamsConfiguration.put(StreamsConfig.STATE_DIR_CONFIG, TestUtils.tempDirectory().getPath() + "/" + storePathSuffix);
+ streamsConfiguration.put(StreamsConfig.STATESTORE_CACHE_MAX_BYTES_CONFIG, 0);
+ streamsConfiguration.put(StreamsConfig.COMMIT_INTERVAL_MS_CONFIG, 100L);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_KEY_SERDE_CLASS_CONFIG, Serdes.LongSerde.class);
+ streamsConfiguration.put(StreamsConfig.DEFAULT_VALUE_SERDE_CLASS_CONFIG, Serdes.StringSerde.class);
+ streamsConfiguration.put(StreamsConfig.NUM_STREAM_THREADS_CONFIG, 1);
+ streamsConfiguration.put(StreamsConfig.NUM_STANDBY_REPLICAS_CONFIG, 1);
+
+ return streamsConfiguration;
+ }
+
+ private static class TestRocksDBStore extends RocksDBStore implements CachedStateStore<Bytes, byte[]> {
+
+ private final CountDownLatch recycleLatch;
+ private final CountDownLatch pendingShutdownLatch;
+ private final AtomicInteger initCount;
+ private final AtomicInteger closeCount;
+
+ public TestRocksDBStore(final String name,
+ final CountDownLatch recycleLatch,
+ final CountDownLatch pendingShutdownLatch,
+ final AtomicInteger initCount,
+ final AtomicInteger closeCount) {
+ super(name, "rocksdb");
+ this.recycleLatch = recycleLatch;
+ this.pendingShutdownLatch = pendingShutdownLatch;
+ this.initCount = initCount;
+ this.closeCount = closeCount;
+ }
+
+ @Override
+ public void init(final StateStoreContext stateStoreContext,
+ final StateStore root) {
+ initCount.incrementAndGet();
+ super.init(stateStoreContext, root);
+ }
+
+ @Override
+ public boolean setFlushListener(final CacheFlushListener<Bytes, byte[]> listener,
+ final boolean sendOldValues) {
+ return false;
+ }
+
+ @Override
+ public void flushCache() {
+ }
Review Comment:
as we just implement CachedStateStore, there is no super
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "lucasbru (via GitHub)" <gi...@apache.org>.
lucasbru commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3846506483
> > But I wonder if we should rename "allTasks" to "allInitializedTasks" then?
> > @lucasbru would you mind if I did that in a followup PR?
No problem, it's a minor thing.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19960: Close pending tasks on shutdown. [kafka]
Posted by "Nikita-Shupletsov (via GitHub)" <gi...@apache.org>.
Nikita-Shupletsov commented on PR #21365:
URL: https://github.com/apache/kafka/pull/21365#issuecomment-3842627254
>Using drainPendingTasksToInit sounds like a good idea to me. Not sure if there are any complications though, but it would seem cleaner code wise.
the problem with drain is that we always try to remove the task we are closing from the task registry: https://github.com/Nikita-Shupletsov/kafka/blob/8ee99ee82df2cf89fbb769d26c66395fd3a63761/streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java#L1405
if we drain the pending tasks, there will be no way for us to confirm it ever existed there. so we either need to relax the check in the remove method or have a separate branch for closing pending to init tasks
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org