You are viewing a plain text version of this content. The canonical link for it is here.
Posted to commits@cassandra.apache.org by "Michael Semb Wever (Jira)" <ji...@apache.org> on 2026/09/18 21:01:00 UTC
[jira] [Updated] (CASSANDRA-21696) Request callbacks registered after messaging service shutdown are never expired
[ https://issues.apache.org/jira/browse/CASSANDRA-21696?page=com.atlassian.jira.plugin.system.issuetabpanels:all-tabpanel ]
Michael Semb Wever updated CASSANDRA-21696:
-------------------------------------------
Description:
{{RequestCallbacks.addWithExpiration}} registers a callback whatever the state of the reaper executor. A request that reaches the coordinator after {{MessagingService.shutdown}} has called {{shutdownNow}} leaves an entry that nothing expires, because the reaper is already shut down, so the entry holds its message and its response handler for the life of the process. CASSANDRA-6948 repaired the same collection of callbacks for the 2.0 expiring map, when the reaper could not be restarted after bootstrap or replace.
The graceful path is worse, because it reschedules itself on the same executor while the map is not empty:
{code:java}
void shutdownGracefully()
{
expire();
if (!callbacks.isEmpty())
executor.schedule(this::shutdownGracefully, 100L, MILLISECONDS);
else
executor.shutdownNow();
}
{code}
A steady arrival of new requests keeps the map populated, so the reaper is never shut down, {{awaitTerminationUntil}} throws {{TimeoutException}}, and {{MessagingService.shutdown}} waits out its whole deadline before it reports the timeout.
The patch sets a flag in both shutdown methods, and each {{addWithExpiration}} overload then reports the late request at trace level and registers nothing. A caller is unaffected: the message is not sent either, so the caller ends on its own deadline, as it does today for a callback the reaper expires.
Patch: [mck/CASSANDRA-21696/5.0|https://github.com/thelastpickle/cassandra/tree/mck/CASSANDRA-21696/5.0]
Provenance: [8aa35c6cab|https://github.com/datastax/cassandra/commit/8aa35c6cab8ccd0ce4326fc60f50c2f9c02e4108] by [~mike_tr_adamson]. That commit carries the same flag and guards; this patch names the message id and the peer in the trace line and adds the regression test the commit lacks.
was:
{{RequestCallbacks.addWithExpiration}} registers a callback whatever the state of the reaper executor. A request that reaches the coordinator after {{MessagingService.shutdown}} has called {{shutdownNow}} leaves an entry that nothing expires, because the reaper is already shut down, so the entry holds its message and its response handler for the life of the process. CASSANDRA-6948 repaired the same collection of callbacks for the 2.0 expiring map, when the reaper could not be restarted after bootstrap or replace.
The graceful path is worse, because it reschedules itself on the same executor while the map is not empty:
{code:java}
void shutdownGracefully()
{
expire();
if (!callbacks.isEmpty())
executor.schedule(this::shutdownGracefully, 100L, MILLISECONDS);
else
executor.shutdownNow();
}
{code}
A steady arrival of new requests keeps the map populated, so the reaper is never shut down, {{awaitTerminationUntil}} throws {{TimeoutException}}, and {{MessagingService.shutdown}} waits out its whole deadline before it reports the timeout.
The patch sets a flag in both shutdown methods, and each {{addWithExpiration}} overload then reports the late request at trace level and registers nothing. A caller is unaffected: the message is not sent either, so the caller ends on its own deadline, as it does today for a callback the reaper expires.
Patch: [mck/upstream/callbacks-after-shutdown/5.0|https://github.com/thelastpickle/cassandra/tree/mck/upstream/callbacks-after-shutdown/5.0]
Provenance: [8aa35c6cab|https://github.com/datastax/cassandra/commit/8aa35c6cab8ccd0ce4326fc60f50c2f9c02e4108] by [~mike_tr_adamson]. That commit carries the same flag and guards; this patch names the message id and the peer in the trace line and adds the regression test the commit lacks.
> Request callbacks registered after messaging service shutdown are never expired
> -------------------------------------------------------------------------------
>
> Key: CASSANDRA-21696
> URL: https://issues.apache.org/jira/browse/CASSANDRA-21696
> Project: Apache Cassandra
> Issue Type: Bug
> Components: Messaging/Internode
> Reporter: Michael Semb Wever
> Priority: Normal
> Fix For: 5.0.x, 6.0.x, 7.x
>
>
> {{RequestCallbacks.addWithExpiration}} registers a callback whatever the state of the reaper executor. A request that reaches the coordinator after {{MessagingService.shutdown}} has called {{shutdownNow}} leaves an entry that nothing expires, because the reaper is already shut down, so the entry holds its message and its response handler for the life of the process. CASSANDRA-6948 repaired the same collection of callbacks for the 2.0 expiring map, when the reaper could not be restarted after bootstrap or replace.
> The graceful path is worse, because it reschedules itself on the same executor while the map is not empty:
> {code:java}
> void shutdownGracefully()
> {
> expire();
> if (!callbacks.isEmpty())
> executor.schedule(this::shutdownGracefully, 100L, MILLISECONDS);
> else
> executor.shutdownNow();
> }
> {code}
> A steady arrival of new requests keeps the map populated, so the reaper is never shut down, {{awaitTerminationUntil}} throws {{TimeoutException}}, and {{MessagingService.shutdown}} waits out its whole deadline before it reports the timeout.
> The patch sets a flag in both shutdown methods, and each {{addWithExpiration}} overload then reports the late request at trace level and registers nothing. A caller is unaffected: the message is not sent either, so the caller ends on its own deadline, as it does today for a callback the reaper expires.
> Patch: [mck/CASSANDRA-21696/5.0|https://github.com/thelastpickle/cassandra/tree/mck/CASSANDRA-21696/5.0]
> Provenance: [8aa35c6cab|https://github.com/datastax/cassandra/commit/8aa35c6cab8ccd0ce4326fc60f50c2f9c02e4108] by [~mike_tr_adamson]. That commit carries the same flag and guards; this patch names the message id and the peer in the trace line and adds the regression test the commit lacks.
--
This message was sent by Atlassian Jira
(v8.20.10#820010)
---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org