Migrating ZooKeeper cluster to new hostnames via rolling restart

[Martin Kou <bi...@gmail.com>]

Hi guys,

I'm looking to switch out 2 servers out of a 5-server cluster. The problem
for me is.. the 2 servers will have different IP addresses and host names.

So basically the migration is going to look like this..

Before: H1, H2, H3, H4, H5
After: H6, H7, H3, H4, H5

The ZooKeeper version I'm using is 3.4.x and so I'll need to perform a
rolling restart. For simplicity, let's assume I'd do the rolling restart
from H1 to H5 sequentially.

Now for some reason (don't ask why) .. the system I'm managing would start
the ZooKeeper servers as soon as I put in the new config. So before I'd
even begin the rolling restart, H6 and H7 would get started.

Now, the question is.. would H6 and H7 join the existing quorum when
they're started?

This is important, because if they don't... then it's possible for me to
have a short down time when the rolling restart reaches H3. After H3 is
restarted, then H3, H6 and H7 would form a quorum.

If they do, however.. then there'd be no down time. In that case, there'd
be momentarily a cluster made up of H1, H2, H3, H4, H5, H6, H7 - before I
begin the rolling shut down / restart.

So what's ZooKeeper's behavior in this case?

Best Regards,
Martin Kou