Re: [I] Clustering issues leading to all nodes being downed [pekko]

["fredfp (via GitHub)" <gi...@apache.org>]

fredfp commented on issue #578:
URL: https://github.com/apache/pekko/issues/578#issuecomment-2462051991

   Here's my understanding:
   1. When initially marking an association as quarantined, the `Quarantined` control message is not sent to the remote when harmless is true: https://github.com/apache/pekko/blob/8cb7d256dcc1498b79a9fff815146fb5b1f451f0/remote/src/main/scala/org/apache/pekko/remote/artery/Association.scala#L569-L572
   2. now comes `InboundQuarantineCheck` into play (used in `ArteryTransport.inboundSink` and `.inboundControlSink`), it serves 2 purposes: a) drop messages incoming through a quarantined association and b) telling again the remote node it is quarantined using `inboundContext.sendControl(association.remoteAddress, Quarantined(...))` in case it somehow didn't already get the message sent in 1.
   3. when a node [learns it is quarantined](https://github.com/apache/pekko/blob/06467547390c417506990fe0d21f50d6d0ea24cc/remote/src/main/scala/org/apache/pekko/remote/artery/ArteryTransport.scala#L564) as a result of 2.b above, it will trigger the SBR to down itself via `ThisActorSystemQuarantinedEvent`, and this is what brings the whole cluster down.
   4. the problem is triggered by 2.b above, which sends `Quarantined` control message also for harmless quarantines, when this case is carefully avoided in 1. We see in `InboundQuarantineCheck` that it doesn't rely on the quarantined status being passed via an event, but instead it is accessed directly via `env.association` and `association.associationState.isQuarantined()`. At this stage however, we lost whether the quarantine was harmless or not. This extra flag should be kept in the `Association` state so that it can be recovered in `InboundQuarantineCheck`.
   
   About reproducing, I'm not sure because it's not clear to me how a node, when shutting down, can quarantine associations to others with `harmless=true`. However, if that can be done I'd suggest:
   - start a cluster with 2 nodes A, B.
   - shutdown A such that it quarantines the association to B with `harmless=true`
   - send messages from B to A, this should trigger `InboundQuarantineCheck` in A to send `Quarantined` to B (and B shutting down as a result), leading the whole cluster to be down.


-- 
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org

For queries about this service, please contact Infrastructure at:
users@infra.apache.org


---------------------------------------------------------------------
To unsubscribe, e-mail: notifications-unsubscribe@pekko.apache.org
For additional commands, e-mail: notifications-help@pekko.apache.org