You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by GitBox <gi...@apache.org> on 2021/05/11 05:24:56 UTC
[GitHub] [kafka] ableegoldman opened a new pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
ableegoldman opened a new pull request #10666:
URL: https://github.com/apache/kafka/pull/10666
Currently `KafkaStreams#cleanUp` only throw an IllegalStateException if the state is RUNNING or REBALANCING, however the application could be in the process of shutting down in which case StreamThreads may still be running. We should also throw if the state is PENDING_ERROR or PENDING_SHUTDOWN
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632106662
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
I think if they call a non blocking close then clean up then later notice the leftovers that is not a great experience and if can prevent it we should. I agree probably not worth a KIP though
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632038675
##########
File path: streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java
##########
@@ -298,7 +300,9 @@ private void prepareStreams() throws Exception {
);
}
- private void prepareStreamThread(final StreamThread thread, final int threadId, final boolean terminable) throws Exception {
+ private void prepareStreamThread(final StreamThread thread,
Review comment:
Yeah, sorry for the confusion
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632021643
##########
File path: streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java
##########
@@ -298,7 +300,9 @@ private void prepareStreams() throws Exception {
);
}
- private void prepareStreamThread(final StreamThread thread, final int threadId, final boolean terminable) throws Exception {
+ private void prepareStreamThread(final StreamThread thread,
Review comment:
I have no problem leaving them on new lines, just didn't understand why when it didn't change the contents
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632065843
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
I already have a unit test in this PR for the `PENDING` case, I should be able to adapt that to the new behavior without much trouble, if we want to change it. I also don't think it would be that difficult to implement this behavior -- I guess it was more a question of whether we should handle this semi-transparently or just tell the user to be patient and try again after the close. For example, at the moment `close()` will launch the shutdown helper as a daemon thread, so if the user does a non-blocking close followed by `cleanUp` and that's the end of the app, most likely the clean up will not occur (nor will the app complete its shutdown)
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632040863
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
This test failing does give me some small pause on just throwing an IllegalStateException in case of PENDING_SHUTDOWN/ERROR -- do we think many users follow this pattern of calling the non-blocking `close` and then immediately calling `cleanUp()`? Maybe the better question is, should we support this, or is it alright to just enforce that users wait for Streams to shut down before invoking `cleanUp` -- if we do want to support this, we could just signal to the shutdown thread that it should re-invoke `cleanUp` once Streams finished shutting down, instead of just throwing an IllegalStateException. Not sure if it's really worth it or not, though -- thoughts @guozhangwang @wcarlson5 ?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r631997442
##########
File path: streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java
##########
@@ -298,7 +300,9 @@ private void prepareStreams() throws Exception {
);
}
- private void prepareStreamThread(final StreamThread thread, final int threadId, final boolean terminable) throws Exception {
+ private void prepareStreamThread(final StreamThread thread,
Review comment:
No, I think this was just left over from when I had added a new parameter which put it over 100, but I took it out when I found a better way to do the test.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#issuecomment-840888170
Just some unrelated flaky tests:
`kafka.network.DynamicConnectionQuotaTest.testDynamicListenerConnectionCreationRateQuota()`
`kafka.connect.integration.ConnectorClientPolicyIntegrationTest.testCreateWithOverridesForNonePolicy()`
`kafka.server.RaftClusterTest.testCreateClusterAndCreateListDeleteTopic()`
`kafka.server.RaftClusterTest.testCreateClusterAndCreateAndManyTopicsWithManyPartitions()`
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632047274
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
How hard could it be to have `cleanUp()` check set a flag for the shutdown thread to call clean up after it's done? We want to make sure the shutdown thread checks after it changes the state to `NOT_RUNNING` or `ERROR`.
I think this a much more flexible pattern that would be good to support if it's running or rebalancing we would still want to throw and `IllegalStateException` or maybe `IllegalOperationException`. But either way it would be good to test when stream is in a` PENDING` state. But I think you would have to make that a unit test
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman merged pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman merged pull request #10666:
URL: https://github.com/apache/kafka/pull/10666
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632066989
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
I guess another alternative is just to have `cleanUp` block until Streams finishes shutting down, but I don't think we want to make it essentially block on the results of `close()`, at least not without giving users a way to supply a timeout, and I don't think it's worth doing an entire KIP just to add an overload with one
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] guozhangwang commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
guozhangwang commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r632117934
##########
File path: streams/src/main/java/org/apache/kafka/streams/processor/internals/StateDirectory.java
##########
@@ -413,43 +413,31 @@ private void cleanRemovedTasksCalledByCleanerThread(final long cleanupDelayMs) {
}
}
- private void cleanRemovedTasksCalledByUser() throws Exception {
+ private void cleanStateAndTaskDirectoriesCalledByUser() throws Exception {
+ if (!lockedTasksToOwner.isEmpty()) {
+ log.warn("Found some still-locked task directories when user requested to cleaning up the state, "
+ + "since Streams is not running any more these will be ignored to complete the cleanup");
+ }
final AtomicReference<Exception> firstException = new AtomicReference<>();
for (final File taskDir : listAllTaskDirectories()) {
final String dirName = taskDir.getName();
final TaskId id = TaskId.parseTaskDirectoryName(dirName, null);
- if (!lockedTasksToOwner.containsKey(id)) {
- try {
- if (lock(id)) {
- log.info("{} Deleting state directory {} for task {} as user calling cleanup.",
- logPrefix(), dirName, id);
- Utils.delete(taskDir);
- } else {
- log.warn("{} Could not get lock for state directory {} for task {} as user calling cleanup.",
- logPrefix(), dirName, id);
- }
- } catch (final OverlappingFileLockException | IOException exception) {
- log.error(
- String.format("%s Failed to delete state directory %s for task %s with exception:",
- logPrefix(), dirName, id),
- exception
- );
- firstException.compareAndSet(null, exception);
- } finally {
- try {
- unlock(id);
- // for manual user call, stream threads are not running so it is safe to delete
- // the whole directory
- Utils.delete(taskDir);
- } catch (final IOException exception) {
- log.error(
- String.format("%s Failed to release lock on state directory %s for task %s with exception:",
- logPrefix(), dirName, id),
- exception
- );
- firstException.compareAndSet(null, exception);
- }
+ try {
+ log.info("{} Deleting state directory {} for task {} as user calling cleanup.",
+ logPrefix(), dirName, id);
+
+ if (lockedTasksToOwner.containsKey(id)) {
+ log.warn("{} Task {} in state directory {} was still locked by {}",
+ logPrefix(), dirName, id, lockedTasksToOwner.get(id));
}
+ Utils.delete(taskDir);
Review comment:
Ah that's a nice one: we are sure all threads are gone now so there's no need to lock :)
##########
File path: streams/src/test/java/org/apache/kafka/streams/integration/StandbyTaskEOSIntegrationTest.java
##########
@@ -170,8 +170,8 @@ public void shouldSurviveWithOneTaskAsStandby() throws Exception {
// Wait for the record to be processed
assertTrue(instanceLatch.await(15, TimeUnit.SECONDS));
- streamInstanceOne.close(Duration.ZERO);
- streamInstanceTwo.close(Duration.ZERO);
Review comment:
At the moment, if user calls a close() with no blocking and then cleanUp, the latter `cleanUp` could become a no-op because the locks are not released yet, and user won't know; whereas now it would throw an exception saying that close is not done yet, right? Plus our javadoc of `cleanUp` states that
```
* Do a clean up of the local {@link StateStore} directory ({@link StreamsConfig#STATE_DIR_CONFIG}) by deleting all
* data with regard to the {@link StreamsConfig#APPLICATION_ID_CONFIG application ID}.
* <p>
* May only be called either before this {@code KafkaStreams} instance is {@link #start() started} or after the
* instance is {@link #close() closed}.
```
Hence I think the new behavior is not really a regression but really a fix forward.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] wcarlson5 commented on a change in pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
wcarlson5 commented on a change in pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#discussion_r631954181
##########
File path: streams/src/test/java/org/apache/kafka/streams/KafkaStreamsTest.java
##########
@@ -298,7 +300,9 @@ private void prepareStreams() throws Exception {
);
}
- private void prepareStreamThread(final StreamThread thread, final int threadId, final boolean terminable) throws Exception {
+ private void prepareStreamThread(final StreamThread thread,
Review comment:
was this over 100?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
[GitHub] [kafka] ableegoldman commented on pull request #10666: MINOR: prevent cleanup() from being called while Streams is still shutting down
Posted by GitBox <gi...@apache.org>.
ableegoldman commented on pull request #10666:
URL: https://github.com/apache/kafka/pull/10666#issuecomment-840173582
cc @guozhangwang @cadonna @wcarlson5 @lct45 to review
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
For queries about this service, please contact Infrastructure at:
users@infra.apache.org