[jira] [Created] (IGNITE-18605) Account for inherent unreliability of messaging

["Roman Puchkovskiy (Jira)" <ji...@apache.org>]

Roman Puchkovskiy created IGNITE-18605:
------------------------------------------

             Summary: Account for inherent unreliability of messaging
                 Key: IGNITE-18605
                 URL: https://issues.apache.org/jira/browse/IGNITE-18605
             Project: Ignite
          Issue Type: Improvement
          Components: networking
            Reporter: Roman Puchkovskiy
             Fix For: 3.0.0-beta2


We use ScaleCube for discovery. It uses SWIM protocol that relies on timeuts. This means that for some reason a node might not send a ping/pong message timely (for example, if it encounters a long JVM pause), which results in other nodes thinking that it has disappeared.

First, we should carefully choose the default values for timeouts: if they are too low, the probability of dropping a node from a cluster is very high (but if they are too high, some tests might take a lot longer).

Also, we should account for the possibility of a node to be dropped from the cluster. This means that:
 # When we get a node from a physical topology, we must always check that it was returned and handle 'no such node in the topology' gracefully
 # Long message exchanges (for instance, streaming a RAFT snapshot) must be robust so as to survive short disappearances of nodes (it would be bad to waste a snapshot installation that already took 30 minutes just because of a transient failure)
 # We need to avoid hangs in cases if someone waits for a message that never gets delivered due to such a transient failure (could this be the cause for IGNITE-18506, where sometimes a message never arrives to an internal cursor?)



--
This message was sent by Atlassian Jira
(v8.20.10#820010)