[jira] [Created] (CASSANDRA-21428) Nodes can become stuck in DOWN if ECHO_REQ has timeout

["Cameron Zemek (Jira)" <ji...@apache.org>]

Cameron Zemek created CASSANDRA-21428:
-----------------------------------------

             Summary: Nodes can become stuck in DOWN if ECHO_REQ has timeout
                 Key: CASSANDRA-21428
                 URL: https://issues.apache.org/jira/browse/CASSANDRA-21428
             Project: Apache Cassandra
          Issue Type: Bug
          Components: Cluster/Gossip
            Reporter: Cameron Zemek


In Gossiper, echoHandler only implements onResponse. RequestCallback.onFailure has a default no-op, so when the ECHO_REQ times out or the remote node returns an error, inflightEcho.remove(addr) is never called. The stale entry persists. Any subsequent markAlive(addr, localState) call — where localState is the same in-place-mutated object already in inflightEcho — sees localState.equals(prevState) = true (identity equality, same reference) and skips indefinitely. In a temporary-partition scenario (node briefly unreachable, echo times out, node recovers with the same generation), the node can get stuck permanently dead: the failure detector sees it as alive and keeps triggering markAlive, but every invocation is suppressed by the stale entry. The stale entry is only cleared by removeEndpoint() (explicit removal) or silentlyMarkDead() via markDead() (failure detector conviction) — neither of which fires if the failure detector is reporting the node as healthy.

Fix: override onFailure in echoHandler to call inflightEcho.remove(addr).
{code:java}
diff --git a/src/java/org/apache/cassandra/gms/Gossiper.java b/src/java/org/apache/cassandra/gms/Gossiper.java
index 647441ffba..6d046cd8ba 100644
--- a/src/java/org/apache/cassandra/gms/Gossiper.java
+++ b/src/java/org/apache/cassandra/gms/Gossiper.java
@@ -65,6 +65,7 @@ import org.apache.cassandra.config.CassandraRelevantProperties;
 import org.apache.cassandra.config.DatabaseDescriptor;
 import org.apache.cassandra.db.SystemKeyspace;
 import org.apache.cassandra.dht.Token;
+import org.apache.cassandra.exceptions.RequestFailureReason;
 import org.apache.cassandra.locator.InetAddressAndPort;
 import org.apache.cassandra.net.Message;
 import org.apache.cassandra.net.MessagingService;
@@ -1442,13 +1443,30 @@ public class Gossiper implements IFailureDetectionEventListener, GossiperMBean,
         {
             Message<NoPayload> echoMessage = Message.out(ECHO_REQ, noPayload);
             logger.trace("Sending ECHO_REQ to {}", addr);
-            RequestCallback echoHandler = msg ->
+            RequestCallback echoHandler = new RequestCallback()
             {
-                runInGossipStageBlocking(() -> {
-                    EndpointState epState = inflightEcho.remove(addr);
-                    if (epState != null)
-                        realMarkAlive(addr, epState);
-                });
+                @Override
+                public void onResponse(Message msg)
+                {
+                    runInGossipStageBlocking(() -> {
+                        EndpointState epState = inflightEcho.remove(addr);
+                        if (epState != null)
+                            realMarkAlive(addr, epState);
+                    });
+                }
+
+                @Override
+                public boolean invokeOnFailure()
+                {
+                    return true;
+                }
+
+                @Override
+                public void onFailure(InetAddressAndPort from, RequestFailureReason failureReason)
+                {
+                    logger.trace("ECHO_REQ to {} failed ({})", addr, failureReason);
+                    inflightEcho.remove(addr);
+                }
             };
             MessagingService.instance().sendWithCallback(echoMessage, addr, echoHandler);
         } {code}



--
This message was sent by Atlassian Jira
(v8.20.10#820010)

---------------------------------------------------------------------
To unsubscribe, e-mail: commits-unsubscribe@cassandra.apache.org
For additional commands, e-mail: commits-help@cassandra.apache.org