[GitHub] [pulsar] devinbost commented on issue #8333: Bookie returning from being off for several days resulted in split-brain

[GitBox <gi...@apache.org>]

devinbost commented on issue #8333:
URL: https://github.com/apache/pulsar/issues/8333#issuecomment-714723497


   "Split-brain" meaning all but one of the bookies suddenly couldn't connect to the cluster. 
   We tried to resolve the problem by shutting down all the bookies that couldn't connect. 
   Then, we provisioned new bookies. They were able to successfully join the cluster with the one that was still up.
   Then, after about an hour, we tried starting one of the bookies that hadn't been able to join. 
   Almost immediately after we did that, almost all of the bookies (even the ones that were newly provisioned) stopped being able to connect to the cluster. We tried different combinations of restarts, but we weren't able to resolve the issue.
   
   The environment is Pulsar 2.4.0 running bare metal on docker. The docker containers are run as systemd services on CentOS boxes. 


----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.

For queries about this service, please contact Infrastructure at:
users@infra.apache.org