You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by David Song <he...@gmail.com> on 2012/04/17 08:48:02 UTC
How to replace a zookeeper server ?
hi,
I have three zookeeper servers, such as A/B/C online. But now I want to
replace box C with a new box named D. What's the best way to update the
server group?
Re: How to replace a zookeeper server ?
Posted by Camille Fournier <ca...@apache.org>.
Agreed that just restarting is probably the easiest way to go. I'm
trying to reason how to do this with a rolling restart without any
downtime at all, and I'm not 100% sure that is safely possible.
Start with A B C up
Reconfigure A to be A B D
Restart A. Quorum is still satisfied.
Take down C. A and B are still up with quorum.
At this point you could bring up D. I need one of the better experts
on the leader election protocol to answer what happens then. B will
not recognize D as a quorum member. If B is leading, D won't get into
the quorum. If A is leading, D will get into the quorum (I think).
Either way, if you then bring down B, A and D should find each other
and be able to form a quorum. Then restart B with A B D and all is
well. The outstanding question in my mind is whether it's safe to have
D try to join when one of the quorum members knows about it but the
other doesn't.
C
On Wed, Apr 18, 2012 at 2:04 AM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
> David, does zookeeper need to be up (i.e. no quorum loss) while you
> update the server list? If not, it's probably the easiest to shut down
> the entire cluster and bring up the new cluster.
>
> Camille, what's the order of rolling restart that doesn't cause quorum
> loss? Would it cause any issues to have inconsistent server list
> during this process?
>
> Thanks!
> --Michi
>
> On Tue, Apr 17, 2012 at 7:50 PM, Camille Fournier <ca...@apache.org> wrote:
>> Probably a variant on the rolling restart is the right way to go:
>> http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A6
>>
>>
>> On Tue, Apr 17, 2012 at 2:48 AM, David Song <he...@gmail.com> wrote:
>>> hi,
>>> I have three zookeeper servers, such as A/B/C online. But now I want to
>>> replace box C with a new box named D. What's the best way to update the
>>> server group?
Re: How to replace a zookeeper server ?
Posted by Jordan Zimmerman <jz...@netflix.com>.
FYI - making this process more reliable is one of the main reasons for
Exhibitor:
http://techblog.netflix.com/2012/04/introducing-exhibitor-supervisor-system
.html
On 4/18/12 8:44 AM, "Ted Dunning" <te...@gmail.com> wrote:
>As long as the old quorum constitutes a quorum in the new cluster you
>should be fine.
>
>In the replace a dead node scenario you can reconfigure the survivors
>before bringing up the new node. That only works if you start with 5
>nodes, but being down a node in a three node cluster is a problem for any
>admin actions you might take.
>
>Sent from my iPhone
>
>On Apr 17, 2012, at 11:04 PM, Michi Mutsuzaki <mi...@cs.stanford.edu>
>wrote:
>
>> Camille, what's the order of rolling restart that doesn't cause quorum
>> loss? Would it cause any issues to have inconsistent server list
>> during this process?
>
Re: How to replace a zookeeper server ?
Posted by Ted Dunning <te...@gmail.com>.
As long as the old quorum constitutes a quorum in the new cluster you should be fine.
In the replace a dead node scenario you can reconfigure the survivors before bringing up the new node. That only works if you start with 5 nodes, but being down a node in a three node cluster is a problem for any admin actions you might take.
Sent from my iPhone
On Apr 17, 2012, at 11:04 PM, Michi Mutsuzaki <mi...@cs.stanford.edu> wrote:
> Camille, what's the order of rolling restart that doesn't cause quorum
> loss? Would it cause any issues to have inconsistent server list
> during this process?
Re: How to replace a zookeeper server ?
Posted by Michi Mutsuzaki <mi...@cs.stanford.edu>.
David, does zookeeper need to be up (i.e. no quorum loss) while you
update the server list? If not, it's probably the easiest to shut down
the entire cluster and bring up the new cluster.
Camille, what's the order of rolling restart that doesn't cause quorum
loss? Would it cause any issues to have inconsistent server list
during this process?
Thanks!
--Michi
On Tue, Apr 17, 2012 at 7:50 PM, Camille Fournier <ca...@apache.org> wrote:
> Probably a variant on the rolling restart is the right way to go:
> http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A6
>
>
> On Tue, Apr 17, 2012 at 2:48 AM, David Song <he...@gmail.com> wrote:
>> hi,
>> I have three zookeeper servers, such as A/B/C online. But now I want to
>> replace box C with a new box named D. What's the best way to update the
>> server group?
Re: How to replace a zookeeper server ?
Posted by David Song <he...@gmail.com>.
Great thanks!
在 2012年4月18日 上午10:50，Camille Fournier <ca...@apache.org>写道：
> Probably a variant on the rolling restart is the right way to go:
> http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A6
>
>
> On Tue, Apr 17, 2012 at 2:48 AM, David Song <he...@gmail.com> wrote:
> > hi,
> > I have three zookeeper servers, such as A/B/C online. But now I want to
> > replace box C with a new box named D. What's the best way to update the
> > server group?
>
Re: How to replace a zookeeper server ?
Posted by Camille Fournier <ca...@apache.org>.
Probably a variant on the rolling restart is the right way to go:
http://wiki.apache.org/hadoop/ZooKeeper/FAQ#A6
On Tue, Apr 17, 2012 at 2:48 AM, David Song <he...@gmail.com> wrote:
> hi,
> I have three zookeeper servers, such as A/B/C online. But now I want to
> replace box C with a new box named D. What's the best way to update the
> server group?