You are viewing a plain text version of this content. The canonical link for it is here.
Posted to jira@kafka.apache.org by "kamalcph (via GitHub)" <gi...@apache.org> on 2025/08/12 10:21:30 UTC
[PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
kamalcph opened a new pull request, #20342:
URL: https://github.com/apache/kafka/pull/20342
During shutdown, when the RSM closes first, then the ongoing requests might throw an error. To handle the ongoing requests gracefully, closing the RSM after closing the remote-log reader thread pools.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290589077
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
Review Comment:
@kamalcph thanks for your explanation.
> The RemoteStorageReaderThreadPool which handles the client RemoteFetch requests also waits for 10 seconds. Shall we bump this to 30 seconds?
It should be fine if the interrupted requests are client fetches. I assumed they might also include some data or metadata sync operations.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2292089211
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
open #20394
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph merged PR #20342:
URL: https://github.com/apache/kafka/pull/20342
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2287635937
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
Review Comment:
If the purpose of this PR is to ensure the requests are handled gracefully, should we also consider increasing the wait time?
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
Could you please add comments explaining the new close order? Otherwise, it might be unintentionally changed in the future
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290429757
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
Followed caller -> callee pattern. RemoteIndexCache invokes remoteStorageManagerPlugin to fetch the indexes so changed the shutdown order.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290963237
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
thanks for confirmation. We will file a minor PR for it.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290417448
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
Review Comment:
The wait time is set to 10 seconds. The [ThreadUtils#shutdownExecutorServiceQuietly](https://sourcegraph.com/github.com/apache/kafka/-/blob/clients/src/main/java/org/apache/kafka/common/utils/ThreadUtils.java?L95) waits twice during `shutdown` and `shutdownNow` operations.
The copy/expiration/follower tasks were already cancelled with interrupt set to true. Those are broker background operations. We can also make it graceful but it can delay the server shutdown time if the remote storage is slow.
The RemoteStorageReaderThreadPool which handles the client RemoteFetch requests also waits for 10 seconds. Shall we bump this to 30 seconds?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290659047
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
seems to be a miss to me.
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "kamalcph (via GitHub)" <gi...@apache.org>.
kamalcph commented on PR #20342:
URL: https://github.com/apache/kafka/pull/20342#issuecomment-3180270521
@satishd @showuon
Call for review. PTAL. Thanks!
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org
Re: [PR] KAFKA-19597: Stop the RSM after closing the remote-log reader threads to handle requests gracefully [kafka]
Posted by "chia7712 (via GitHub)" <gi...@apache.org>.
chia7712 commented on code in PR #20342:
URL: https://github.com/apache/kafka/pull/20342#discussion_r2290612736
##########
storage/src/main/java/org/apache/kafka/server/log/remote/storage/RemoteLogManager.java:
##########
@@ -2050,10 +2047,13 @@ public void close() {
} finally {
removeMetrics();
}
-
leaderCopyRLMTasks.clear();
leaderExpirationRLMTasks.clear();
followerRLMTasks.clear();
+
+ Utils.closeQuietly(indexCache, "RemoteIndexCache");
Review Comment:
> Followed caller -> callee pattern. RemoteIndexCache invokes remoteStorageManagerPlugin to fetch the indexes so changed the shutdown order.
yes, that is good pattern. BTW, it looks like `fetchQuotaMetrics` and `copyQuotaMetrics` are excluded by the `removeMetrics`. Is that expected?
--
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
To unsubscribe, e-mail: jira-unsubscribe@kafka.apache.org
For queries about this service, please contact Infrastructure at:
users@infra.apache.org