You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2022/08/29 17:13:29 UTC
[GitHub] [kafka] cadonna opened a new pull request, #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
cadonna opened a new pull request, #12569:
URL: https://github.com/apache/kafka/pull/12569
When the task manager is shutdown, the state updater should also
shutdown. After the shutdown of the state updater, the tasks
in its output queues should be closed.
### Committer Checklist (excluded from commit message)
- [ ] Verify design and implementation
- [ ] Verify test coverage and CI build status
- [ ] Verify documentation (including upgrade notes)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna merged pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
cadonna merged PR #12569:
URL: https://github.com/apache/kafka/pull/12569
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a diff in pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on code in PR #12569:
URL: https://github.com/apache/kafka/pull/12569#discussion_r958716921
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1180,6 +1182,68 @@ void shutdown(final boolean clean) {
}
}
+ private void shutdownStateUpdater() {
+ if (stateUpdater != null) {
+ stateUpdater.shutdown(Duration.ofMillis(Long.MAX_VALUE));
Review Comment:
This is a meta thought: I'm not sure if passing in Long.MAX_VALUE is the best thing to do. When we shutdown a stream thread we do have a timeout at the shutdown thread waiting for those threads to join, and I'm thinking maybe we should propagate that timeout from StreamThread.completeShutdown -> TaskManager -> StateUpdater (and similarly for all other modules of a stream thread that we are shutting down).
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1180,6 +1182,68 @@ void shutdown(final boolean clean) {
}
}
+ private void shutdownStateUpdater() {
+ if (stateUpdater != null) {
+ stateUpdater.shutdown(Duration.ofMillis(Long.MAX_VALUE));
+ closeFailedTasks();
+ addRestoredTasksToTaskRegistry();
+ addRemovedTasksToTaskRegistry();
+ }
+ }
+
+ private void closeFailedTasks() {
+ final Set<Task> tasksToCloseDirty = stateUpdater.drainExceptionsAndFailedTasks().stream()
+ .flatMap(exAndTasks -> exAndTasks.getTasks().stream()).collect(Collectors.toSet());
+
+ for (final Task task : tasksToCloseDirty) {
+ try {
+ // we call this function only to flush the case if necessary
+ // before suspending and closing the topology
+ task.prepareCommit();
+ } catch (final RuntimeException swallow) {
+ log.error("Error flushing caches of dirty task {} ", task.id(), swallow);
+ }
+
+ try {
+ task.suspend();
+ } catch (final RuntimeException swallow) {
+ log.error("Error suspending dirty task {}: {}", task.id(), swallow.getMessage());
+ }
+
+ task.closeDirty();
+
+ try {
+ if (task.isActive()) {
+ activeTaskCreator.closeAndRemoveTaskProducerIfNeeded(task.id());
+ }
+ } catch (final RuntimeException swallow) {
+ log.error("Error closing dirty task {}: {}", task.id(), swallow.getMessage());
+ }
+ }
+ }
+
+ private void addRestoredTasksToTaskRegistry() {
Review Comment:
Again, here I'm assuming we are going to merge the two lists and hence their handling logic later, so did not focus too much on it.
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1180,6 +1182,68 @@ void shutdown(final boolean clean) {
}
}
+ private void shutdownStateUpdater() {
+ if (stateUpdater != null) {
+ stateUpdater.shutdown(Duration.ofMillis(Long.MAX_VALUE));
+ closeFailedTasks();
+ addRestoredTasksToTaskRegistry();
+ addRemovedTasksToTaskRegistry();
+ }
+ }
+
+ private void closeFailedTasks() {
Review Comment:
nit: closeFailedTasksFromStateUpdater?
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1180,6 +1182,68 @@ void shutdown(final boolean clean) {
}
}
+ private void shutdownStateUpdater() {
+ if (stateUpdater != null) {
+ stateUpdater.shutdown(Duration.ofMillis(Long.MAX_VALUE));
+ closeFailedTasks();
+ addRestoredTasksToTaskRegistry();
+ addRemovedTasksToTaskRegistry();
+ }
+ }
+
+ private void closeFailedTasks() {
Review Comment:
Also I have some thoughts about error/failed tasks handling in general, but let's table it for now and discuss after we've done the exception handling in https://github.com/apache/kafka/pull/12519.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12569:
URL: https://github.com/apache/kafka/pull/12569#issuecomment-1230599261
Call for review: @wcarlson5 @lihaosky
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12569:
URL: https://github.com/apache/kafka/pull/12569#issuecomment-1233290152
@guozhangwang As discussed yesterday, I will merge this PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
cadonna commented on PR #12569:
URL: https://github.com/apache/kafka/pull/12569#issuecomment-1233289415
Build failures are unrelated:
```
Build / JDK 11 and Scala 2.13 / kafka.api.TransactionsExpirationTest.testTransactionAfterProducerIdExpires(String).quorum=zk
Build / JDK 8 and Scala 2.12 / org.apache.kafka.clients.consumer.CooperativeStickyAssignorTest.testLargeAssignmentAndGroupWithNonEqualSubscription()
Build / JDK 8 and Scala 2.12 / org.apache.kafka.clients.consumer.StickyAssignorTest.testLargeAssignmentAndGroupWithNonEqualSubscription()
Build / JDK 8 and Scala 2.12 / kafka.api.PlaintextAdminIntegrationTest.testConsumerGroups(String).quorum=zk
```
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] cadonna commented on a diff in pull request #12569: KAFKA-10199: Shutdown state updater on task manager shutdown
Posted by GitBox <gi...@apache.org>.
cadonna commented on code in PR #12569:
URL: https://github.com/apache/kafka/pull/12569#discussion_r959907923
##########
streams/src/main/java/org/apache/kafka/streams/processor/internals/TaskManager.java:
##########
@@ -1180,6 +1182,68 @@ void shutdown(final boolean clean) {
}
}
+ private void shutdownStateUpdater() {
+ if (stateUpdater != null) {
+ stateUpdater.shutdown(Duration.ofMillis(Long.MAX_VALUE));
+ closeFailedTasks();
+ addRestoredTasksToTaskRegistry();
+ addRemovedTasksToTaskRegistry();
+ }
+ }
+
+ private void closeFailedTasks() {
Review Comment:
I will do the renaming in the next PR.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org