Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

Hi,

We are testing integration with 3.5.0 and dynamic membership and I have a
question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
dynamic config file on servers F,G,H be configured on startup?  Should they
have the old ensemble, the new ensemble, or the union of both ensembles?
 It seems like these new servers need to  know about the old quorum, but
since they aren't part of it yet its not clear to me how they should be
configured.  Should there be an intermediate configuration with F,G, and H
as simply Observers?

I can't find much documentation on this so I want to make sure I understand
things correctly.

Thanks!
~Jared

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

BTW, please don't hesitate to ask me if you have other questions or run
into any issues with ZK-107 patch.

On Fri, Jul 27, 2012 at 5:20 PM, Alexander Shraer <sh...@gmail.com> wrote:

> Hi Jared,
>
> Thanks for experimenting with this feature.
>
> The idea is that new servers join as "non voting followers". Which means
> that they act as normal followers but the leader ignores their votes since
> they are not part of the current configuration. The leader only counts
> their votes during the reconfiguration itself (to make sure a quorum of the
> new config is ready before the new config can be committed/activated).
> Defining them as observers is not a good idea, for example in your scenario
> if they were observers they wouldn't be able to participate in the
> reconfiguration protocol (which is similar to the protocol for committing
> any other operation in which observers don't participate) and since we
> don't have a quorum of followers in the new config that can ack,
> reconfiguration would throw an exception (of
> KeeperException.NEWCONFIGNOQUORUM type).
> Of course if you intend them to be observers in the new config you can
> define them as observers since their votes are not needed during reconfig
> anyway.
>
> You're right, the new servers must be able to connect to the old quorum.
> At minimum, their file should contain the current leader, but
> you can also copy the current configuration file to the new members if you
> wish.
>
> In addition, you should add a line for the member itself, so that server F
> appears in F's config file (Its not important that the other new servers
> appear in F's file, but it won't hurt either, so you can do a union of old
> and new if you wish). The constructor of QuorumPeer checks that the server
> itself is in the configuration its started with, otherwise its not going to
> run. This check has always been there, but I'm thinking of possibly
> changing it in the future.
>
> As soon as F connects to the leader, its config file will be overwritten
> with the current config file as part of the synchronization process.
>
> Alex
>
>
> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <jared.cantwell@gmail.com
> > wrote:
>
>> Hi,
>>
>> We are testing integration with 3.5.0 and dynamic membership and I have a
>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>> dynamic config file on servers F,G,H be configured on startup?  Should
>> they
>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>  It seems like these new servers need to  know about the old quorum, but
>> since they aren't part of it yet its not clear to me how they should be
>> configured.  Should there be an intermediate configuration with F,G, and H
>> as simply Observers?
>>
>> I can't find much documentation on this so I want to make sure I
>> understand
>> things correctly.
>>
>> Thanks!
>> ~Jared
>>
>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

And a followup... does zookeeper only overwrite the dynamic configuration
file for nodes that are voting participants?  Such that if I started a
follower and then left it running through some reconfigurations, its file
would not get updated if it was never added as part of those
reconfigurations?

~Jared

On Fri, Jul 27, 2012 at 6:35 PM, Jared Cantwell <ja...@gmail.com>wrote:

> So does just having the server started and pointing to the existing
> ensemble automatically make it a "non participating follower"?  In other
> words, there is no need to inform the existing nodes that this new node is
> joining as a follower?  And to extend that, there could be any number of
> followers that are simply listening in on the event stream?  I am assuming
> that's the case, and that it is a follower (and not participant) by virtue
> of not being in the official configuration stored in zookeeper itself.
>
> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>
>> there are just two supported types - participant and observer.
>> (participant can act as either follower or leader).
>>
>> So you can either write participant or leave it unspecified (which means
>> participant by default). Also, since the ip is the same for all your ports
>> you don't have to write it twice.  All of these should work in the same way:
>>
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183:participant;2181 <http://10.10.5.17:2181/>
>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>
>>
>>
>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <jared.cantwell@gmail.com
>> > wrote:
>>
>>> Thanks Alex for the response.  Our current lines in the configuration
>>> look like this:
>>>
>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>
>>> For the new servers is it ok for their entry to have "participant"?  Or
>>> should that be something different (e.g. "follower")?
>>>
>>> ~Jared
>>>
>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>
>>>> Hi Jared,
>>>>
>>>> Thanks for experimenting with this feature.
>>>>
>>>> The idea is that new servers join as "non voting followers". Which
>>>> means that they act as normal followers but the leader ignores their votes
>>>> since they are not part of the current configuration. The leader only
>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>> of the new config is ready before the new config can be
>>>> committed/activated). Defining them as observers is not a good idea, for
>>>> example in your scenario if they were observers they wouldn't be able to
>>>> participate in the reconfiguration protocol (which is similar to the
>>>> protocol for committing any other operation in which observers don't
>>>> participate) and since we don't have a quorum of followers in the new
>>>> config that can ack, reconfiguration would throw an exception (of
>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>> Of course if you intend them to be observers in the new config you can
>>>> define them as observers since their votes are not needed during reconfig
>>>> anyway.
>>>>
>>>> You're right, the new servers must be able to connect to the old
>>>> quorum. At minimum, their file should contain the current leader, but
>>>> you can also copy the current configuration file to the new members if
>>>> you wish.
>>>>
>>>> In addition, you should add a line for the member itself, so that
>>>> server F appears in F's config file (Its not important that the other new
>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>> server itself is in the configuration its started with, otherwise its not
>>>> going to run. This check has always been there, but I'm thinking of
>>>> possibly changing it in the future.
>>>>
>>>> As soon as F connects to the leader, its config file will be
>>>> overwritten with the current config file as part of the synchronization
>>>> process.
>>>>
>>>> Alex
>>>>
>>>>
>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>> jared.cantwell@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>> have a
>>>>> question.  If I have a current set of servers in my ensemble
>>>>> {A,B,C,D,E}
>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>>>> dynamic config file on servers F,G,H be configured on startup?  Should
>>>>> they
>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>> ensembles?
>>>>>  It seems like these new servers need to  know about the old quorum,
>>>>> but
>>>>> since they aren't part of it yet its not clear to me how they should be
>>>>> configured.  Should there be an intermediate configuration with F,G,
>>>>> and H
>>>>> as simply Observers?
>>>>>
>>>>> I can't find much documentation on this so I want to make sure I
>>>>> understand
>>>>> things correctly.
>>>>>
>>>>> Thanks!
>>>>> ~Jared
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

Yes, this way two new guys will not connect to each other because they dont know about each other. Every quorum including a bew guy must include at least one old guy that knows the latest config and state. For example if the current config is ABCDE, its possible that D and never heard any ops. If we're adding F then FDE is not a majority of ABCDEF there must be at least one that knows all commited state. This is true only for majority quorums (which is the one being used by probably everyone). For hierarchical quorums I'd use the meathod of leader+self. 

Alex

Sent from mobile

On Jul 28, 2012, at 4:52 PM, Jared Cantwell <ja...@gmail.com> wrote:

> I really liked the idea of just starting the config with the new server and the leader of the old config, but I want to make sure I understand your latest statement.  You're concluding that if a server were to start up at a very later time with a config of old+self then we'd be safe still.  I think that is because the only way a quorum can be established in this case is if it includes a majority of the old config members, but those servers should all (at least) know of a newer configuration (if they're even still around), and they would be refusing to start unless the leader was in that new configuration.  So as long as a given configuration doesn't ever have a majority of servers that weren't in the config at some point then we're good.  This of course makes sense (and has always been a terrible terrible thing to do), but that can easily be avoided by never doing a union of old+new.
> 
> ~Jared
> 
> On Sat, Jul 28, 2012 at 5:36 PM, Alexander Shraer <sh...@gmail.com> wrote:
> On second thought, your scenario would not be possible even if you initiate a server with the current config plus only itself.
> I agree that you should not do union of old and new configs.
> Alex
> 
> Sent from mobile
> 
> On Jul 28, 2012, at 3:26 PM, Jared Cantwell <ja...@gmail.com> wrote:
> 
>> Having to "make up" a configuration for new servers that are non voting followers is something I keep getting stuck on, because if a couple of these servers start with just the wrong configuration they can all have an unspecified version and actually form a standalone quorum and not know a newer one exists. I would feel much better if new servers that are to become non voting followers could simply copy the configuration file from an existing voting participant (including  the version) and still start up. This is particularly an issue for us because servers with arbitrarily old configurations can start up at any time, and if the wrong things happen then they could form a quorum if they have the right "bootstrap" configurations with unspecified versions. 
>> 
>> Does this make sense as a concern? We may patch the server to allow starting up without its myid in the configuration. 
>> 
>> Jared 
>> 
>> On Jul 28, 2012, at 12:15 PM, Alexander Shraer <sh...@gmail.com> wrote:
>> 
>>> yes, if the server reboots (this is when it would read the config file). 
>>> Otherwise, it has the last config in memory (this is held in a QuorumVerifier object in QuorumPeer) and it doesn't look in the config file. 
>>> 
>>> BTW the config file (when overwritten by the system) has an auto-generated version using which we know which config is later than which.  Users are not supposed to specify this version at all - its supposed to be managed by the system. If you replace the file and set the version to something low or not specify it at all, chances are that the config file will be overwritten during synchronization with the leader or during communication with other servers in FastLeaderElection.
>>> If you set it to something high, its possible that your server will be able to convince others that this is the latest config :)
>>> 
>>> Alex
>>> 
>>> 
>>> On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell <ja...@gmail.com> wrote:
>>> No that you would want to do this, but simply overwriting a config file would "uncommit" a configuration and make that server think the last committed configuration was whatever is in the file?
>>> 
> 

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

I really liked the idea of just starting the config with the new server and
the leader of the old config, but I want to make sure I understand your
latest statement.  You're concluding that if a server were to start up at a
very later time with a config of old+self then we'd be safe still.  I think
that is because the only way a quorum can be established in this case is if
it includes a majority of the old config members, but those servers should
all (at least) know of a newer configuration (if they're even still
around), and they would be refusing to start unless the leader was in that
new configuration.  So as long as a given configuration doesn't *ever* have
a majority of servers that weren't in the config at some point then we're
good.  This of course makes sense (and has always been a terrible terrible
thing to do), but that can easily be avoided by never doing a union of
old+new.

~Jared

On Sat, Jul 28, 2012 at 5:36 PM, Alexander Shraer <sh...@gmail.com> wrote:

> On second thought, your scenario would not be possible even if you
> initiate a server with the current config plus only itself.
> I agree that you should not do union of old and new configs.
> Alex
>
> Sent from mobile
>
> On Jul 28, 2012, at 3:26 PM, Jared Cantwell <ja...@gmail.com>
> wrote:
>
> Having to "make up" a configuration for new servers that are non voting
> followers is something I keep getting stuck on, because if a couple of
> these servers start with just the wrong configuration they can all have an
> unspecified version and actually form a standalone quorum and not know a
> newer one exists. I would feel much better if new servers that are to
> become non voting followers could simply copy the configuration file from
> an existing voting participant (including  the version) and still start up.
> This is particularly an issue for us because servers with arbitrarily old
> configurations can start up at any time, and if the wrong things happen
> then they could form a quorum if they have the right "bootstrap"
> configurations with unspecified versions.
>
> Does this make sense as a concern? We may patch the server to allow
> starting up without its myid in the configuration.
>
> Jared
>
> On Jul 28, 2012, at 12:15 PM, Alexander Shraer <sh...@gmail.com> wrote:
>
> yes, if the server reboots (this is when it would read the config file).
> Otherwise, it has the last config in memory (this is held in a
> QuorumVerifier object in QuorumPeer) and it doesn't look in the config
> file.
>
> BTW the config file (when overwritten by the system) has an auto-generated
> version using which we know which config is later than which.  Users are
> not supposed to specify this version at all - its supposed to be managed by
> the system. If you replace the file and set the version to something low or
> not specify it at all, chances are that the config file will be overwritten
> during synchronization with the leader or during communication with other
> servers in FastLeaderElection.
> If you set it to something high, its possible that your server will be
> able to convince others that this is the latest config :)
>
> Alex
>
>
> On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell <jared.cantwell@gmail.com
> > wrote:
>
>> No that you would want to do this, but simply overwriting a config file
>> would "uncommit" a configuration and make that server think the last
>> committed configuration was whatever is in the file?
>>
>
>

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

On second thought, your scenario would not be possible even if you initiate a server with the current config plus only itself.
I agree that you should not do union of old and new configs.
Alex

Sent from mobile

On Jul 28, 2012, at 3:26 PM, Jared Cantwell <ja...@gmail.com> wrote:

> Having to "make up" a configuration for new servers that are non voting followers is something I keep getting stuck on, because if a couple of these servers start with just the wrong configuration they can all have an unspecified version and actually form a standalone quorum and not know a newer one exists. I would feel much better if new servers that are to become non voting followers could simply copy the configuration file from an existing voting participant (including  the version) and still start up. This is particularly an issue for us because servers with arbitrarily old configurations can start up at any time, and if the wrong things happen then they could form a quorum if they have the right "bootstrap" configurations with unspecified versions. 
> 
> Does this make sense as a concern? We may patch the server to allow starting up without its myid in the configuration. 
> 
> Jared 
> 
> On Jul 28, 2012, at 12:15 PM, Alexander Shraer <sh...@gmail.com> wrote:
> 
>> yes, if the server reboots (this is when it would read the config file). 
>> Otherwise, it has the last config in memory (this is held in a QuorumVerifier object in QuorumPeer) and it doesn't look in the config file. 
>> 
>> BTW the config file (when overwritten by the system) has an auto-generated version using which we know which config is later than which.  Users are not supposed to specify this version at all - its supposed to be managed by the system. If you replace the file and set the version to something low or not specify it at all, chances are that the config file will be overwritten during synchronization with the leader or during communication with other servers in FastLeaderElection.
>> If you set it to something high, its possible that your server will be able to convince others that this is the latest config :)
>> 
>> Alex
>> 
>> 
>> On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell <ja...@gmail.com> wrote:
>> No that you would want to do this, but simply overwriting a config file would "uncommit" a configuration and make that server think the last committed configuration was whatever is in the file?
>> 

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

I suggest that you start each server with only itself and the current leader in the config file. I agree that we should find a better way like simply starting it with the current config file. Its not that simple to do though since a server needs certain parameters like one of the ports to communicate with others. I'll look into this, but cant do it in the coming week.

Alex

Sent from mobile

On Jul 28, 2012, at 3:26 PM, Jared Cantwell <ja...@gmail.com> wrote:

> Having to "make up" a configuration for new servers that are non voting followers is something I keep getting stuck on, because if a couple of these servers start with just the wrong configuration they can all have an unspecified version and actually form a standalone quorum and not know a newer one exists. I would feel much better if new servers that are to become non voting followers could simply copy the configuration file from an existing voting participant (including  the version) and still start up. This is particularly an issue for us because servers with arbitrarily old configurations can start up at any time, and if the wrong things happen then they could form a quorum if they have the right "bootstrap" configurations with unspecified versions. 
> 
> Does this make sense as a concern? We may patch the server to allow starting up without its myid in the configuration. 
> 
> Jared 
> 
> On Jul 28, 2012, at 12:15 PM, Alexander Shraer <sh...@gmail.com> wrote:
> 
>> yes, if the server reboots (this is when it would read the config file). 
>> Otherwise, it has the last config in memory (this is held in a QuorumVerifier object in QuorumPeer) and it doesn't look in the config file. 
>> 
>> BTW the config file (when overwritten by the system) has an auto-generated version using which we know which config is later than which.  Users are not supposed to specify this version at all - its supposed to be managed by the system. If you replace the file and set the version to something low or not specify it at all, chances are that the config file will be overwritten during synchronization with the leader or during communication with other servers in FastLeaderElection.
>> If you set it to something high, its possible that your server will be able to convince others that this is the latest config :)
>> 
>> Alex
>> 
>> 
>> On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell <ja...@gmail.com> wrote:
>> No that you would want to do this, but simply overwriting a config file would "uncommit" a configuration and make that server think the last committed configuration was whatever is in the file?
>> 

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

Having to "make up" a configuration for new servers that are non voting followers is something I keep getting stuck on, because if a couple of these servers start with just the wrong configuration they can all have an unspecified version and actually form a standalone quorum and not know a newer one exists. I would feel much better if new servers that are to become non voting followers could simply copy the configuration file from an existing voting participant (including  the version) and still start up. This is particularly an issue for us because servers with arbitrarily old configurations can start up at any time, and if the wrong things happen then they could form a quorum if they have the right "bootstrap" configurations with unspecified versions. 

Does this make sense as a concern? We may patch the server to allow starting up without its myid in the configuration. 

Jared 

On Jul 28, 2012, at 12:15 PM, Alexander Shraer <sh...@gmail.com> wrote:

> yes, if the server reboots (this is when it would read the config file). 
> Otherwise, it has the last config in memory (this is held in a QuorumVerifier object in QuorumPeer) and it doesn't look in the config file. 
> 
> BTW the config file (when overwritten by the system) has an auto-generated version using which we know which config is later than which.  Users are not supposed to specify this version at all - its supposed to be managed by the system. If you replace the file and set the version to something low or not specify it at all, chances are that the config file will be overwritten during synchronization with the leader or during communication with other servers in FastLeaderElection.
> If you set it to something high, its possible that your server will be able to convince others that this is the latest config :)
> 
> Alex
> 
> 
> On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell <ja...@gmail.com> wrote:
> No that you would want to do this, but simply overwriting a config file would "uncommit" a configuration and make that server think the last committed configuration was whatever is in the file?
> 

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

yes, if the server reboots (this is when it would read the config file).
Otherwise, it has the last config in memory (this is held in a
QuorumVerifier object in QuorumPeer) and it doesn't look in the config
file.

BTW the config file (when overwritten by the system) has an auto-generated
version using which we know which config is later than which.  Users are
not supposed to specify this version at all - its supposed to be managed by
the system. If you replace the file and set the version to something low or
not specify it at all, chances are that the config file will be overwritten
during synchronization with the leader or during communication with other
servers in FastLeaderElection.
If you set it to something high, its possible that your server will be able
to convince others that this is the latest config :)

Alex


On Sat, Jul 28, 2012 at 10:57 AM, Jared Cantwell
<ja...@gmail.com>wrote:

> No that you would want to do this, but simply overwriting a config file
> would "uncommit" a configuration and make that server think the last
> committed configuration was whatever is in the file?
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

No that you would want to do this, but simply overwriting a config file would "uncommit" a configuration and make that server think the last committed configuration was whatever is in the file?

Jared

On Jul 28, 2012, at 11:33 AM, Alexander Shraer <sh...@gmail.com> wrote:

> No problem! 
> 
> The way it works is that before a server acks a reconfig operation it writes a special tmp file to disk (dynamicConfigFilename + ".tmp"). Servers look for this file during recovery, they don't look for the configuration in the log as for normal data, because we found it to be difficult to extract the right info from the log exactly at the stage we needed it in the recovery. When a commit message is received a server renames the tmp file to dynamicConfigFilename.
> 
> Recently there was a change committed by someone to start using atomic file operations for different files in ZooKeeper. At some point we'll probably change the renaming above to use these atomic operations.
> 
> Alex
>   
> 
> On Sat, Jul 28, 2012 at 10:17 AM, Jared Cantwell <ja...@gmail.com> wrote:
> Thanks Alex for the detailed explanations--  it really helps to fill in my understanding of the implementation left open by the papers/presentations I've read (without having to read the code yet :-) ).  #2 is what I was unsure of, but makes perfect sense.
> 
> Obviously committing the new configuration to the internal database is a prerequisite to committing on a server, but is writing the new configuration file to disk also a prerequisite for committing the new configuration?  I'm curious about this so I can match it with my observations, since reading the configuration file is much easier than getting the database state.
> 
> ~Jared
> 
> 
> On Sat, Jul 28, 2012 at 11:02 AM, Alexander Shraer <sh...@gmail.com> wrote:
> Hi Jared,
> 
> figuring out what happened and how to recover is part of the reconfiguration protocol. I don't think that this is something you as a user should do, unless I missunderstand what you're trying to do. This should be handled by ZooKeeper just like it handles other failures without admin intervention. 
> 
> In your scenario, D-F come up and one of them is elected leader (since you said they know about the commit), so they start running the new config normally. When A-C come up, several things may happen: 
> 
> 1. During the preliminary FastLeaderElection, A-C will try to connect to D and E, and in fact they'll also try to connect with the new config members that they know was proposed. So most chances are that someone in the new config will send them the new config file and they'll store it and act accordingly (connect as non-voting followers in the new config). To make this happen, I changed FastLeaderElection to talk with proposed configs (if known) and to piggiback the last active config you know of on all messages.
> 
> 2. Its possible that somehow A-C complete FastLeaderElection without talking to D-F. But since a reconfiguration was committed, it was acked by a quorum of the old config (and a quorum of the new one). Therefore, whoever is "elected" in the old config, knows about the reconfig proposal (this is guaranteed by normal ZooKeeper leader recovery). Before doing anything else, the new leader among A-C will try to complete the reconfiguration, which involves getting enough acks from a quorum of the new config. But in your scenario the servers in the new config will not connect to it because they moved on, so the candidate-leader will just give up and go back to (1) above. 
> 
> 3. In the remote chance that someone who heard about the reconfig commit connects to a candidate-leader who didn't hear about it, the first thing it does  is to tell that candidate-leader that its not up to date, and the leader just updates its config file, gives up on being a leader and returns to (1). This was done by changing the first message that a follower/observer sends to a leader it is connecting to, even before the synchronization starts.
> 
> Alex
> 
> 
> 
> On Sat, Jul 28, 2012 at 8:43 AM, Jared Cantwell  <ja...@gmail.com> wrote:
> So I'm working through some failure scenarios and I want to make sure I fully understand the way that dynamic membership changes previous behavior, so are my expectations correct in this situation:
> 
> As in my previous example, lets say that the current membership of voting participants is {A,B,C,D,E} and we're looking to change membership to {D,E,F,G,H}. 
> 1. Reconfiguration to {D,E,F,G,H} completes internally 
> 2. D-F update their local configuration files, but A-C do not yet.
> 3. Power loss to all nodes
> 
> Now what happens if A,B, and C come up with configuration files that still say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C form a quorum and elect a leader since they all agree on the same state?  What then happens when the new membership of D-H starts up?
> 
> We're trying to automatically handle node failures during reconfiguration situations, but it seems like without being able to query all nodes to make sure you know of the latest membership list there is no safe way to do this.  I'm wondering if only doing single node additions/removals would create less complicated failure scenarios.  What are your thoughts and best practices around this?
> 
> Thanks!
> Jared
> 
> On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <ja...@gmail.com> wrote:
> We are trying to remove the need for all admin intervention so that is one failure scenario that is interesting to us. 
> 
> Jared
> 
> 
> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com> wrote:
> 
>> Yes, this entry will be deleted. I don't like this either - if a new follower reboots before added to the config it will not be able to boot up without manual help from an admin. That's why I'm considering maybe to remove the check that a participant must always initially be in its own config, but for now its there.
>> 
>> Alex
>> 
>> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <ja...@gmail.com> wrote:
>> Sorry for the confusion in terminology, I was unfamiliar with the exact leader/follower semantics previously. 
>> 
>> So if all connected servers update their config file, does that mean that non-voting followers who aren't part of the new ensemble will lose the entry specific to them in their config file?  I can test this myself, but getting an inside perspective is very helpful. 
>> 
>> Thanks again for the help!
>> Jared
>> 
>> 
>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>> 
>>> Yes, any number of followers which are not in the configuration can just connect and listen in. This has always been the case, also in 3.4, I just made use of this for the purpose of adding members during reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113
>>> where the leader actually counts the votes of anyone connected, regardless of config membership :) This is fixed in ZK-107, so they are really non-voting followers. 
>>> 
>>> >   I am assuming that's the case, and that it is a follower (and not 
>>> > participant) by virtue of not being in the official configuration stored in 
>>> > zookeeper itself. 
>>> 
>>> Follower and participant types of servers is not something that was defined in ZK-107. In ZooKeeper every follower/leader is a "participant". Its just that the votes of participants that are not in the configuration are not counted that's why we call them non-voting followers. BTW, obviously a non-voting follower can not become leader (like ZK-1113 this was also not enforced before ZK-107).
>>> 
>>> > And a followup... does zookeeper only overwrite the dynamic 
>>> > configuration file for nodes that are voting participants?  Such that if I 
>>> > started a follower and then left it running through some 
>>> > reconfigurations, its file would not get updated if it was never added as 
>>> > part of those reconfigurations?
>>> 
>>> No, as soon as it connects to the current leader, its dynamic config file is overwritten with the current configuration as part of the synchronization with the leader. Every time a new configuration is committed, all connected servers (voting, non-voting, observers) will update their dynamic config file, doesn't matter if they're in the config.
>>> 
>>> Alex
>>> 
>>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <ja...@gmail.com> wrote:
>>> So does just having the server started and pointing to the existing ensemble automatically make it a "non participating follower"?  In other words, there is no need to inform the existing nodes that this new node is joining as a follower?  And to extend that, there could be any number of followers that are simply listening in on the event stream?  I am assuming that's the case, and that it is a follower (and not participant) by virtue of not being in the official configuration stored in zookeeper itself.
>>> 
>>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>> there are just two supported types - participant and observer.
>>> (participant can act as either follower or leader).
>>> 
>>> So you can either write participant or leave it unspecified (which means participant by default). Also, since the ip is the same for all your ports you don't have to write it twice.  All of these should work in the same way:
>>> 
>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>> server.5=10.10.5.17:2182:2183:participant;2181
>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>> server.5=10.10.5.17:2182:2183;2181
>>> 
>>> 
>>> 
>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <ja...@gmail.com> wrote:
>>> Thanks Alex for the response.  Our current lines in the configuration look like this:
>>> 
>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>> 
>>> For the new servers is it ok for their entry to have "participant"?  Or should that be something different (e.g. "follower")?
>>> 
>>> ~Jared
>>> 
>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>> Hi Jared,
>>> 
>>> Thanks for experimenting with this feature. 
>>> 
>>> The idea is that new servers join as "non voting followers". Which means that they act as normal followers but the leader ignores their votes since they are not part of the current configuration. The leader only counts their votes during the reconfiguration itself (to make sure a quorum of the new config is ready before the new config can be committed/activated). Defining them as observers is not a good idea, for example in your scenario if they were observers they wouldn't be able to participate in the reconfiguration protocol (which is similar to the protocol for committing any other operation in which observers don't participate) and since we don't have a quorum of followers in the new config that can ack, reconfiguration would throw an exception (of KeeperException.NEWCONFIGNOQUORUM type). 
>>> Of course if you intend them to be observers in the new config you can define them as observers since their votes are not needed during reconfig anyway.
>>> 
>>> You're right, the new servers must be able to connect to the old quorum. At minimum, their file should contain the current leader, but 
>>> you can also copy the current configuration file to the new members if you wish. 
>>> 
>>> In addition, you should add a line for the member itself, so that server F appears in F's config file (Its not important that the other new servers appear in F's file, but it won't hurt either, so you can do a union of old and new if you wish). The constructor of QuorumPeer checks that the server itself is in the configuration its started with, otherwise its not going to run. This check has always been there, but I'm thinking of possibly changing it in the future.
>>> 
>>> As soon as F connects to the leader, its config file will be overwritten with the current config file as part of the synchronization process. 
>>> 
>>> Alex
>>> 
>>> 
>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <ja...@gmail.com> wrote:
>>> Hi,
>>> 
>>> We are testing integration with 3.5.0 and dynamic membership and I have a
>>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>> dynamic config file on servers F,G,H be configured on startup?  Should they
>>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>>  It seems like these new servers need to  know about the old quorum, but
>>> since they aren't part of it yet its not clear to me how they should be
>>> configured.  Should there be an intermediate configuration with F,G, and H
>>> as simply Observers?
>>> 
>>> I can't find much documentation on this so I want to make sure I understand
>>> things correctly.
>>> 
>>> Thanks!
>>> ~Jared
>>> 
>>> 
>>> 
>>> 
>>> 
>> 
> 
> 
> 
> 

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

No problem!

The way it works is that before a server acks a reconfig operation it
writes a special tmp file to disk (dynamicConfigFilename + ".tmp"). Servers
look for this file during recovery, they don't look for the configuration
in the log as for normal data, because we found it to be difficult to
extract the right info from the log exactly at the stage we needed it in
the recovery. When a commit message is received a server renames the tmp
file to dynamicConfigFilename.

Recently there was a change committed by someone to start using atomic file
operations for different files in ZooKeeper. At some point we'll probably
change the renaming above to use these atomic operations.

Alex


On Sat, Jul 28, 2012 at 10:17 AM, Jared Cantwell
<ja...@gmail.com>wrote:

> Thanks Alex for the detailed explanations--  it really helps to fill in my
> understanding of the implementation left open by the papers/presentations
> I've read (without having to read the code yet :-) ).  #2 is what I was
> unsure of, but makes perfect sense.
>
> Obviously committing the new configuration to the internal database is a
> prerequisite to committing on a server, but is writing the new *configuration
> file* to disk also a prerequisite for committing the new configuration?
>  I'm curious about this so I can match it with my observations, since
> reading the configuration file is much easier than getting the database
> state.
>
> ~Jared
>
>
> On Sat, Jul 28, 2012 at 11:02 AM, Alexander Shraer <sh...@gmail.com>wrote:
>
>> Hi Jared,
>>
>> figuring out what happened and how to recover is part of the
>> reconfiguration protocol. I don't think that this is something you as a
>> user should do, unless I missunderstand what you're trying to do. This
>> should be handled by ZooKeeper just like it handles other failures without
>> admin intervention.
>>
>> In your scenario, D-F come up and one of them is elected leader (since
>> you said they know about the commit), so they start running the new config
>> normally. When A-C come up, several things may happen:
>>
>> 1. During the preliminary FastLeaderElection, A-C will try to connect to
>> D and E, and in fact they'll also try to connect with the new config
>> members that they know was proposed. So most chances are that someone in
>> the new config will send them the new config file and they'll store it and
>> act accordingly (connect as non-voting followers in the new config). To
>> make this happen, I changed FastLeaderElection to talk with proposed
>> configs (if known) and to piggiback the last active config you know of on
>> all messages.
>>
>> 2. Its possible that somehow A-C complete FastLeaderElection without
>> talking to D-F. But since a reconfiguration was committed, it was acked by
>> a quorum of the old config (and a quorum of the new one). Therefore,
>> whoever is "elected" in the old config, knows about the reconfig proposal
>> (this is guaranteed by normal ZooKeeper leader recovery). Before doing
>> anything else, the new leader among A-C will try to complete the
>> reconfiguration, which involves getting enough acks from a quorum of the
>> new config. But in your scenario the servers in the new config will not
>> connect to it because they moved on, so the candidate-leader will just give
>> up and go back to (1) above.
>>
>> 3. In the remote chance that someone who heard about the reconfig commit
>> connects to a candidate-leader who didn't hear about it, the first thing it
>> does  is to tell that candidate-leader that its not up to date, and the
>> leader just updates its config file, gives up on being a leader and returns
>> to (1). This was done by changing the first message that a
>> follower/observer sends to a leader it is connecting to, even before the
>> synchronization starts.
>>
>> Alex
>>
>>
>>
>> On Sat, Jul 28, 2012 at 8:43 AM, Jared Cantwell  <
>> jared.cantwell@gmail.com> wrote:
>>
>>> So I'm working through some failure scenarios and I want to make sure I
>>> fully understand the way that dynamic membership changes previous behavior,
>>> so are my expectations correct in this situation:
>>>
>>> As in my previous example, lets say that the current membership of
>>> voting participants is {A,B,C,D,E} and we're looking to change membership
>>> to {D,E,F,G,H}.
>>> 1. Reconfiguration to {D,E,F,G,H} completes internally
>>> 2. D-F update their local configuration files, but A-C do not yet.
>>> 3. Power loss to all nodes
>>>
>>> Now what happens if A,B, and C come up with configuration files that
>>> still say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C
>>> form a quorum and elect a leader since they all agree on the same state?
>>>  What then happens when the new membership of D-H starts up?
>>>
>>> We're trying to automatically handle node failures during
>>> reconfiguration situations, but it seems like without being able to query
>>> all nodes to make sure you know of the latest membership list there is no
>>> safe way to do this.  I'm wondering if only doing single node
>>> additions/removals would create less complicated failure scenarios.  What
>>> are your thoughts and best practices around this?
>>>
>>> Thanks!
>>> Jared
>>>
>>> On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <
>>> jared.cantwell@gmail.com> wrote:
>>>
>>>> We are trying to remove the need for all admin intervention so that is
>>>> one failure scenario that is interesting to us.
>>>>
>>>> Jared
>>>>
>>>>
>>>> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com>
>>>> wrote:
>>>>
>>>> Yes, this entry will be deleted. I don't like this either - if a new
>>>> follower reboots before added to the config it will not be able to boot up
>>>> without manual help from an admin. That's why I'm considering maybe to
>>>> remove the check that a participant must always initially be in its own
>>>> config, but for now its there.
>>>>
>>>> Alex
>>>>
>>>> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <
>>>> jared.cantwell@gmail.com> wrote:
>>>>
>>>>> Sorry for the confusion in terminology, I was unfamiliar with the
>>>>> exact leader/follower semantics previously.
>>>>>
>>>>> So if all connected servers update their config file, does that mean
>>>>> that non-voting followers who aren't part of the new ensemble will lose the
>>>>> entry specific to them in their config file?  I can test this myself, but
>>>>> getting an inside perspective is very helpful.
>>>>>
>>>>> Thanks again for the help!
>>>>> Jared
>>>>>
>>>>>
>>>>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com>
>>>>> wrote:
>>>>>
>>>>> Yes, any number of followers which are not in the configuration can
>>>>> just connect and listen in. This has always been the case, also in 3.4, I
>>>>> just made use of this for the purpose of adding members during
>>>>> reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
>>>>> where the leader actually counts the votes of anyone connected,
>>>>> regardless of config membership :) This is fixed in ZK-107, so they are
>>>>> really non-voting followers.
>>>>>
>>>>> >   I am assuming that's the case, and that it is a follower (and not
>>>>> > participant) by virtue of not being in the official configuration
>>>>> stored in
>>>>> > zookeeper itself.
>>>>>
>>>>> Follower and participant types of servers is not something that was
>>>>> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
>>>>> Its just that the votes of participants that are not in the configuration
>>>>> are not counted that's why we call them non-voting followers. BTW,
>>>>> obviously a non-voting follower can not become leader (like ZK-1113 this
>>>>> was also not enforced before ZK-107).
>>>>>
>>>>> > And a followup... does zookeeper only overwrite the dynamic
>>>>> > configuration file for nodes that are voting participants?  Such
>>>>> that if I
>>>>> > started a follower and then left it running through some
>>>>> > reconfigurations, its file would not get updated if it was never
>>>>> added as
>>>>> > part of those reconfigurations?
>>>>>
>>>>> No, as soon as it connects to the current leader, its dynamic config
>>>>> file is overwritten with the current configuration as part of the
>>>>> synchronization with the leader. Every time a new configuration is
>>>>> committed, all connected servers (voting, non-voting, observers) will
>>>>> update their dynamic config file, doesn't matter if they're in the config.
>>>>>
>>>>> Alex
>>>>>
>>>>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <
>>>>> jared.cantwell@gmail.com> wrote:
>>>>>
>>>>>> So does just having the server started and pointing to the existing
>>>>>> ensemble automatically make it a "non participating follower"?  In other
>>>>>> words, there is no need to inform the existing nodes that this new node is
>>>>>> joining as a follower?  And to extend that, there could be any number of
>>>>>> followers that are simply listening in on the event stream?  I am assuming
>>>>>> that's the case, and that it is a follower (and not participant) by virtue
>>>>>> of not being in the official configuration stored in zookeeper itself.
>>>>>>
>>>>>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>>>
>>>>>>> there are just two supported types - participant and observer.
>>>>>>> (participant can act as either follower or leader).
>>>>>>>
>>>>>>> So you can either write participant or leave it unspecified (which
>>>>>>> means participant by default). Also, since the ip is the same for all your
>>>>>>> ports you don't have to write it twice.  All of these should work in the
>>>>>>> same way:
>>>>>>>
>>>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>>> server.5=10.10.5.17:2182:2183:participant;2181<http://10.10.5.17:2181/>
>>>>>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>>>>>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <
>>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>>
>>>>>>>> Thanks Alex for the response.  Our current lines in the
>>>>>>>> configuration look like this:
>>>>>>>>
>>>>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>>>>
>>>>>>>> For the new servers is it ok for their entry to have "participant"?
>>>>>>>>  Or should that be something different (e.g. "follower")?
>>>>>>>>
>>>>>>>> ~Jared
>>>>>>>>
>>>>>>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <
>>>>>>>> shralex@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi Jared,
>>>>>>>>>
>>>>>>>>> Thanks for experimenting with this feature.
>>>>>>>>>
>>>>>>>>> The idea is that new servers join as "non voting followers". Which
>>>>>>>>> means that they act as normal followers but the leader ignores their votes
>>>>>>>>> since they are not part of the current configuration. The leader only
>>>>>>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>>>>>>> of the new config is ready before the new config can be
>>>>>>>>> committed/activated). Defining them as observers is not a good idea, for
>>>>>>>>> example in your scenario if they were observers they wouldn't be able to
>>>>>>>>> participate in the reconfiguration protocol (which is similar to the
>>>>>>>>> protocol for committing any other operation in which observers don't
>>>>>>>>> participate) and since we don't have a quorum of followers in the new
>>>>>>>>> config that can ack, reconfiguration would throw an exception (of
>>>>>>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>>>>>>> Of course if you intend them to be observers in the new config you
>>>>>>>>> can define them as observers since their votes are not needed during
>>>>>>>>> reconfig anyway.
>>>>>>>>>
>>>>>>>>> You're right, the new servers must be able to connect to the old
>>>>>>>>> quorum. At minimum, their file should contain the current leader, but
>>>>>>>>> you can also copy the current configuration file to the new
>>>>>>>>> members if you wish.
>>>>>>>>>
>>>>>>>>> In addition, you should add a line for the member itself, so that
>>>>>>>>> server F appears in F's config file (Its not important that the other new
>>>>>>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>>>>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>>>>>>> server itself is in the configuration its started with, otherwise its not
>>>>>>>>> going to run. This check has always been there, but I'm thinking of
>>>>>>>>> possibly changing it in the future.
>>>>>>>>>
>>>>>>>>> As soon as F connects to the leader, its config file will be
>>>>>>>>> overwritten with the current config file as part of the synchronization
>>>>>>>>> process.
>>>>>>>>>
>>>>>>>>> Alex
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Hi,
>>>>>>>>>>
>>>>>>>>>> We are testing integration with 3.5.0 and dynamic membership and
>>>>>>>>>> I have a
>>>>>>>>>> question.  If I have a current set of servers in my ensemble
>>>>>>>>>> {A,B,C,D,E}
>>>>>>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should
>>>>>>>>>> the
>>>>>>>>>> dynamic config file on servers F,G,H be configured on startup?
>>>>>>>>>>  Should they
>>>>>>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>>>>>>> ensembles?
>>>>>>>>>>  It seems like these new servers need to  know about the old
>>>>>>>>>> quorum, but
>>>>>>>>>> since they aren't part of it yet its not clear to me how they
>>>>>>>>>> should be
>>>>>>>>>> configured.  Should there be an intermediate configuration with
>>>>>>>>>> F,G, and H
>>>>>>>>>> as simply Observers?
>>>>>>>>>>
>>>>>>>>>> I can't find much documentation on this so I want to make sure I
>>>>>>>>>> understand
>>>>>>>>>> things correctly.
>>>>>>>>>>
>>>>>>>>>> Thanks!
>>>>>>>>>> ~Jared
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

Thanks Alex for the detailed explanations--  it really helps to fill in my
understanding of the implementation left open by the papers/presentations
I've read (without having to read the code yet :-) ).  #2 is what I was
unsure of, but makes perfect sense.

Obviously committing the new configuration to the internal database is a
prerequisite to committing on a server, but is writing the new *configuration
file* to disk also a prerequisite for committing the new configuration?
 I'm curious about this so I can match it with my observations, since
reading the configuration file is much easier than getting the database
state.

~Jared

On Sat, Jul 28, 2012 at 11:02 AM, Alexander Shraer <sh...@gmail.com>wrote:

> Hi Jared,
>
> figuring out what happened and how to recover is part of the
> reconfiguration protocol. I don't think that this is something you as a
> user should do, unless I missunderstand what you're trying to do. This
> should be handled by ZooKeeper just like it handles other failures without
> admin intervention.
>
> In your scenario, D-F come up and one of them is elected leader (since you
> said they know about the commit), so they start running the new config
> normally. When A-C come up, several things may happen:
>
> 1. During the preliminary FastLeaderElection, A-C will try to connect to D
> and E, and in fact they'll also try to connect with the new config members
> that they know was proposed. So most chances are that someone in the new
> config will send them the new config file and they'll store it and act
> accordingly (connect as non-voting followers in the new config). To make
> this happen, I changed FastLeaderElection to talk with proposed configs (if
> known) and to piggiback the last active config you know of on all messages.
>
> 2. Its possible that somehow A-C complete FastLeaderElection without
> talking to D-F. But since a reconfiguration was committed, it was acked by
> a quorum of the old config (and a quorum of the new one). Therefore,
> whoever is "elected" in the old config, knows about the reconfig proposal
> (this is guaranteed by normal ZooKeeper leader recovery). Before doing
> anything else, the new leader among A-C will try to complete the
> reconfiguration, which involves getting enough acks from a quorum of the
> new config. But in your scenario the servers in the new config will not
> connect to it because they moved on, so the candidate-leader will just give
> up and go back to (1) above.
>
> 3. In the remote chance that someone who heard about the reconfig commit
> connects to a candidate-leader who didn't hear about it, the first thing it
> does  is to tell that candidate-leader that its not up to date, and the
> leader just updates its config file, gives up on being a leader and returns
> to (1). This was done by changing the first message that a
> follower/observer sends to a leader it is connecting to, even before the
> synchronization starts.
>
> Alex
>
>
>
> On Sat, Jul 28, 2012 at 8:43 AM, Jared Cantwell  <jared.cantwell@gmail.com
> > wrote:
>
>> So I'm working through some failure scenarios and I want to make sure I
>> fully understand the way that dynamic membership changes previous behavior,
>> so are my expectations correct in this situation:
>>
>> As in my previous example, lets say that the current membership of voting
>> participants is {A,B,C,D,E} and we're looking to change membership to
>> {D,E,F,G,H}.
>> 1. Reconfiguration to {D,E,F,G,H} completes internally
>> 2. D-F update their local configuration files, but A-C do not yet.
>> 3. Power loss to all nodes
>>
>> Now what happens if A,B, and C come up with configuration files that
>> still say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C
>> form a quorum and elect a leader since they all agree on the same state?
>>  What then happens when the new membership of D-H starts up?
>>
>> We're trying to automatically handle node failures during reconfiguration
>> situations, but it seems like without being able to query all nodes to make
>> sure you know of the latest membership list there is no safe way to do
>> this.  I'm wondering if only doing single node additions/removals would
>> create less complicated failure scenarios.  What are your thoughts and best
>> practices around this?
>>
>> Thanks!
>> Jared
>>
>> On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <jared.cantwell@gmail.com
>> > wrote:
>>
>>> We are trying to remove the need for all admin intervention so that is
>>> one failure scenario that is interesting to us.
>>>
>>> Jared
>>>
>>>
>>> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>>
>>> Yes, this entry will be deleted. I don't like this either - if a new
>>> follower reboots before added to the config it will not be able to boot up
>>> without manual help from an admin. That's why I'm considering maybe to
>>> remove the check that a participant must always initially be in its own
>>> config, but for now its there.
>>>
>>> Alex
>>>
>>> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <
>>> jared.cantwell@gmail.com> wrote:
>>>
>>>> Sorry for the confusion in terminology, I was unfamiliar with the exact
>>>> leader/follower semantics previously.
>>>>
>>>> So if all connected servers update their config file, does that mean
>>>> that non-voting followers who aren't part of the new ensemble will lose the
>>>> entry specific to them in their config file?  I can test this myself, but
>>>> getting an inside perspective is very helpful.
>>>>
>>>> Thanks again for the help!
>>>> Jared
>>>>
>>>>
>>>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com>
>>>> wrote:
>>>>
>>>> Yes, any number of followers which are not in the configuration can
>>>> just connect and listen in. This has always been the case, also in 3.4, I
>>>> just made use of this for the purpose of adding members during
>>>> reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
>>>> where the leader actually counts the votes of anyone connected,
>>>> regardless of config membership :) This is fixed in ZK-107, so they are
>>>> really non-voting followers.
>>>>
>>>> >   I am assuming that's the case, and that it is a follower (and not
>>>> > participant) by virtue of not being in the official configuration
>>>> stored in
>>>> > zookeeper itself.
>>>>
>>>> Follower and participant types of servers is not something that was
>>>> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
>>>> Its just that the votes of participants that are not in the configuration
>>>> are not counted that's why we call them non-voting followers. BTW,
>>>> obviously a non-voting follower can not become leader (like ZK-1113 this
>>>> was also not enforced before ZK-107).
>>>>
>>>> > And a followup... does zookeeper only overwrite the dynamic
>>>> > configuration file for nodes that are voting participants?  Such that
>>>> if I
>>>> > started a follower and then left it running through some
>>>> > reconfigurations, its file would not get updated if it was never
>>>> added as
>>>> > part of those reconfigurations?
>>>>
>>>> No, as soon as it connects to the current leader, its dynamic config
>>>> file is overwritten with the current configuration as part of the
>>>> synchronization with the leader. Every time a new configuration is
>>>> committed, all connected servers (voting, non-voting, observers) will
>>>> update their dynamic config file, doesn't matter if they're in the config.
>>>>
>>>> Alex
>>>>
>>>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <
>>>> jared.cantwell@gmail.com> wrote:
>>>>
>>>>> So does just having the server started and pointing to the existing
>>>>> ensemble automatically make it a "non participating follower"?  In other
>>>>> words, there is no need to inform the existing nodes that this new node is
>>>>> joining as a follower?  And to extend that, there could be any number of
>>>>> followers that are simply listening in on the event stream?  I am assuming
>>>>> that's the case, and that it is a follower (and not participant) by virtue
>>>>> of not being in the official configuration stored in zookeeper itself.
>>>>>
>>>>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>>
>>>>>> there are just two supported types - participant and observer.
>>>>>> (participant can act as either follower or leader).
>>>>>>
>>>>>> So you can either write participant or leave it unspecified (which
>>>>>> means participant by default). Also, since the ip is the same for all your
>>>>>> ports you don't have to write it twice.  All of these should work in the
>>>>>> same way:
>>>>>>
>>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>> server.5=10.10.5.17:2182:2183:participant;2181<http://10.10.5.17:2181/>
>>>>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>>>>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>>>>>
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <
>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Alex for the response.  Our current lines in the
>>>>>>> configuration look like this:
>>>>>>>
>>>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>>>
>>>>>>> For the new servers is it ok for their entry to have "participant"?
>>>>>>>  Or should that be something different (e.g. "follower")?
>>>>>>>
>>>>>>> ~Jared
>>>>>>>
>>>>>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <shralex@gmail.com
>>>>>>> > wrote:
>>>>>>>
>>>>>>>> Hi Jared,
>>>>>>>>
>>>>>>>> Thanks for experimenting with this feature.
>>>>>>>>
>>>>>>>> The idea is that new servers join as "non voting followers". Which
>>>>>>>> means that they act as normal followers but the leader ignores their votes
>>>>>>>> since they are not part of the current configuration. The leader only
>>>>>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>>>>>> of the new config is ready before the new config can be
>>>>>>>> committed/activated). Defining them as observers is not a good idea, for
>>>>>>>> example in your scenario if they were observers they wouldn't be able to
>>>>>>>> participate in the reconfiguration protocol (which is similar to the
>>>>>>>> protocol for committing any other operation in which observers don't
>>>>>>>> participate) and since we don't have a quorum of followers in the new
>>>>>>>> config that can ack, reconfiguration would throw an exception (of
>>>>>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>>>>>> Of course if you intend them to be observers in the new config you
>>>>>>>> can define them as observers since their votes are not needed during
>>>>>>>> reconfig anyway.
>>>>>>>>
>>>>>>>> You're right, the new servers must be able to connect to the old
>>>>>>>> quorum. At minimum, their file should contain the current leader, but
>>>>>>>> you can also copy the current configuration file to the new members
>>>>>>>> if you wish.
>>>>>>>>
>>>>>>>> In addition, you should add a line for the member itself, so that
>>>>>>>> server F appears in F's config file (Its not important that the other new
>>>>>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>>>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>>>>>> server itself is in the configuration its started with, otherwise its not
>>>>>>>> going to run. This check has always been there, but I'm thinking of
>>>>>>>> possibly changing it in the future.
>>>>>>>>
>>>>>>>> As soon as F connects to the leader, its config file will be
>>>>>>>> overwritten with the current config file as part of the synchronization
>>>>>>>> process.
>>>>>>>>
>>>>>>>> Alex
>>>>>>>>
>>>>>>>>
>>>>>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Hi,
>>>>>>>>>
>>>>>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>>>>>> have a
>>>>>>>>> question.  If I have a current set of servers in my ensemble
>>>>>>>>> {A,B,C,D,E}
>>>>>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should
>>>>>>>>> the
>>>>>>>>> dynamic config file on servers F,G,H be configured on startup?
>>>>>>>>>  Should they
>>>>>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>>>>>> ensembles?
>>>>>>>>>  It seems like these new servers need to  know about the old
>>>>>>>>> quorum, but
>>>>>>>>> since they aren't part of it yet its not clear to me how they
>>>>>>>>> should be
>>>>>>>>> configured.  Should there be an intermediate configuration with
>>>>>>>>> F,G, and H
>>>>>>>>> as simply Observers?
>>>>>>>>>
>>>>>>>>> I can't find much documentation on this so I want to make sure I
>>>>>>>>> understand
>>>>>>>>> things correctly.
>>>>>>>>>
>>>>>>>>> Thanks!
>>>>>>>>> ~Jared
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

Hi Jared,

figuring out what happened and how to recover is part of the
reconfiguration protocol. I don't think that this is something you as a
user should do, unless I missunderstand what you're trying to do. This
should be handled by ZooKeeper just like it handles other failures without
admin intervention.

In your scenario, D-F come up and one of them is elected leader (since you
said they know about the commit), so they start running the new config
normally. When A-C come up, several things may happen:

1. During the preliminary FastLeaderElection, A-C will try to connect to D
and E, and in fact they'll also try to connect with the new config members
that they know was proposed. So most chances are that someone in the new
config will send them the new config file and they'll store it and act
accordingly (connect as non-voting followers in the new config). To make
this happen, I changed FastLeaderElection to talk with proposed configs (if
known) and to piggiback the last active config you know of on all messages.

2. Its possible that somehow A-C complete FastLeaderElection without
talking to D-F. But since a reconfiguration was committed, it was acked by
a quorum of the old config (and a quorum of the new one). Therefore,
whoever is "elected" in the old config, knows about the reconfig proposal
(this is guaranteed by normal ZooKeeper leader recovery). Before doing
anything else, the new leader among A-C will try to complete the
reconfiguration, which involves getting enough acks from a quorum of the
new config. But in your scenario the servers in the new config will not
connect to it because they moved on, so the candidate-leader will just give
up and go back to (1) above.

3. In the remote chance that someone who heard about the reconfig commit
connects to a candidate-leader who didn't hear about it, the first thing it
does  is to tell that candidate-leader that its not up to date, and the
leader just updates its config file, gives up on being a leader and returns
to (1). This was done by changing the first message that a
follower/observer sends to a leader it is connecting to, even before the
synchronization starts.

Alex



On Sat, Jul 28, 2012 at 8:43 AM, Jared Cantwell
<ja...@gmail.com>wrote:

> So I'm working through some failure scenarios and I want to make sure I
> fully understand the way that dynamic membership changes previous behavior,
> so are my expectations correct in this situation:
>
> As in my previous example, lets say that the current membership of voting
> participants is {A,B,C,D,E} and we're looking to change membership to
> {D,E,F,G,H}.
> 1. Reconfiguration to {D,E,F,G,H} completes internally
> 2. D-F update their local configuration files, but A-C do not yet.
> 3. Power loss to all nodes
>
> Now what happens if A,B, and C come up with configuration files that still
> say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C form a
> quorum and elect a leader since they all agree on the same state?  What
> then happens when the new membership of D-H starts up?
>
> We're trying to automatically handle node failures during reconfiguration
> situations, but it seems like without being able to query all nodes to make
> sure you know of the latest membership list there is no safe way to do
> this.  I'm wondering if only doing single node additions/removals would
> create less complicated failure scenarios.  What are your thoughts and best
> practices around this?
>
> Thanks!
> Jared
>
> On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <ja...@gmail.com>wrote:
>
>> We are trying to remove the need for all admin intervention so that is
>> one failure scenario that is interesting to us.
>>
>> Jared
>>
>>
>> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>
>> Yes, this entry will be deleted. I don't like this either - if a new
>> follower reboots before added to the config it will not be able to boot up
>> without manual help from an admin. That's why I'm considering maybe to
>> remove the check that a participant must always initially be in its own
>> config, but for now its there.
>>
>> Alex
>>
>> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <jared.cantwell@gmail.com
>> > wrote:
>>
>>> Sorry for the confusion in terminology, I was unfamiliar with the exact
>>> leader/follower semantics previously.
>>>
>>> So if all connected servers update their config file, does that mean
>>> that non-voting followers who aren't part of the new ensemble will lose the
>>> entry specific to them in their config file?  I can test this myself, but
>>> getting an inside perspective is very helpful.
>>>
>>> Thanks again for the help!
>>> Jared
>>>
>>>
>>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>>
>>> Yes, any number of followers which are not in the configuration can just
>>> connect and listen in. This has always been the case, also in 3.4, I just
>>> made use of this for the purpose of adding members during reconfiguration.
>>> Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
>>> where the leader actually counts the votes of anyone connected,
>>> regardless of config membership :) This is fixed in ZK-107, so they are
>>> really non-voting followers.
>>>
>>> >   I am assuming that's the case, and that it is a follower (and not
>>> > participant) by virtue of not being in the official configuration
>>> stored in
>>> > zookeeper itself.
>>>
>>> Follower and participant types of servers is not something that was
>>> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
>>> Its just that the votes of participants that are not in the configuration
>>> are not counted that's why we call them non-voting followers. BTW,
>>> obviously a non-voting follower can not become leader (like ZK-1113 this
>>> was also not enforced before ZK-107).
>>>
>>> > And a followup... does zookeeper only overwrite the dynamic
>>> > configuration file for nodes that are voting participants?  Such that
>>> if I
>>> > started a follower and then left it running through some
>>> > reconfigurations, its file would not get updated if it was never added
>>> as
>>> > part of those reconfigurations?
>>>
>>> No, as soon as it connects to the current leader, its dynamic config
>>> file is overwritten with the current configuration as part of the
>>> synchronization with the leader. Every time a new configuration is
>>> committed, all connected servers (voting, non-voting, observers) will
>>> update their dynamic config file, doesn't matter if they're in the config.
>>>
>>> Alex
>>>
>>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <
>>> jared.cantwell@gmail.com> wrote:
>>>
>>>> So does just having the server started and pointing to the existing
>>>> ensemble automatically make it a "non participating follower"?  In other
>>>> words, there is no need to inform the existing nodes that this new node is
>>>> joining as a follower?  And to extend that, there could be any number of
>>>> followers that are simply listening in on the event stream?  I am assuming
>>>> that's the case, and that it is a follower (and not participant) by virtue
>>>> of not being in the official configuration stored in zookeeper itself.
>>>>
>>>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>
>>>>> there are just two supported types - participant and observer.
>>>>> (participant can act as either follower or leader).
>>>>>
>>>>> So you can either write participant or leave it unspecified (which
>>>>> means participant by default). Also, since the ip is the same for all your
>>>>> ports you don't have to write it twice.  All of these should work in the
>>>>> same way:
>>>>>
>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>> server.5=10.10.5.17:2182:2183:participant;2181<http://10.10.5.17:2181/>
>>>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>>>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>>>>
>>>>>
>>>>>
>>>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <
>>>>> jared.cantwell@gmail.com> wrote:
>>>>>
>>>>>> Thanks Alex for the response.  Our current lines in the configuration
>>>>>> look like this:
>>>>>>
>>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>>
>>>>>> For the new servers is it ok for their entry to have "participant"?
>>>>>>  Or should that be something different (e.g. "follower")?
>>>>>>
>>>>>> ~Jared
>>>>>>
>>>>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>>>
>>>>>>> Hi Jared,
>>>>>>>
>>>>>>> Thanks for experimenting with this feature.
>>>>>>>
>>>>>>> The idea is that new servers join as "non voting followers". Which
>>>>>>> means that they act as normal followers but the leader ignores their votes
>>>>>>> since they are not part of the current configuration. The leader only
>>>>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>>>>> of the new config is ready before the new config can be
>>>>>>> committed/activated). Defining them as observers is not a good idea, for
>>>>>>> example in your scenario if they were observers they wouldn't be able to
>>>>>>> participate in the reconfiguration protocol (which is similar to the
>>>>>>> protocol for committing any other operation in which observers don't
>>>>>>> participate) and since we don't have a quorum of followers in the new
>>>>>>> config that can ack, reconfiguration would throw an exception (of
>>>>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>>>>> Of course if you intend them to be observers in the new config you
>>>>>>> can define them as observers since their votes are not needed during
>>>>>>> reconfig anyway.
>>>>>>>
>>>>>>> You're right, the new servers must be able to connect to the old
>>>>>>> quorum. At minimum, their file should contain the current leader, but
>>>>>>> you can also copy the current configuration file to the new members
>>>>>>> if you wish.
>>>>>>>
>>>>>>> In addition, you should add a line for the member itself, so that
>>>>>>> server F appears in F's config file (Its not important that the other new
>>>>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>>>>> server itself is in the configuration its started with, otherwise its not
>>>>>>> going to run. This check has always been there, but I'm thinking of
>>>>>>> possibly changing it in the future.
>>>>>>>
>>>>>>> As soon as F connects to the leader, its config file will be
>>>>>>> overwritten with the current config file as part of the synchronization
>>>>>>> process.
>>>>>>>
>>>>>>> Alex
>>>>>>>
>>>>>>>
>>>>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>>
>>>>>>>> Hi,
>>>>>>>>
>>>>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>>>>> have a
>>>>>>>> question.  If I have a current set of servers in my ensemble
>>>>>>>> {A,B,C,D,E}
>>>>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should
>>>>>>>> the
>>>>>>>> dynamic config file on servers F,G,H be configured on startup?
>>>>>>>>  Should they
>>>>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>>>>> ensembles?
>>>>>>>>  It seems like these new servers need to  know about the old
>>>>>>>> quorum, but
>>>>>>>> since they aren't part of it yet its not clear to me how they
>>>>>>>> should be
>>>>>>>> configured.  Should there be an intermediate configuration with
>>>>>>>> F,G, and H
>>>>>>>> as simply Observers?
>>>>>>>>
>>>>>>>> I can't find much documentation on this so I want to make sure I
>>>>>>>> understand
>>>>>>>> things correctly.
>>>>>>>>
>>>>>>>> Thanks!
>>>>>>>> ~Jared
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

So I'm working through some failure scenarios and I want to make sure I
fully understand the way that dynamic membership changes previous behavior,
so are my expectations correct in this situation:

As in my previous example, lets say that the current membership of voting
participants is {A,B,C,D,E} and we're looking to change membership to
{D,E,F,G,H}.
1. Reconfiguration to {D,E,F,G,H} completes internally
2. D-F update their local configuration files, but A-C do not yet.
3. Power loss to all nodes

Now what happens if A,B, and C come up with configuration files that still
say {A,B,C,D,E}, but no other servers start up yet?  Can A,B and C form a
quorum and elect a leader since they all agree on the same state?  What
then happens when the new membership of D-H starts up?

We're trying to automatically handle node failures during reconfiguration
situations, but it seems like without being able to query all nodes to make
sure you know of the latest membership list there is no safe way to do
this.  I'm wondering if only doing single node additions/removals would
create less complicated failure scenarios.  What are your thoughts and best
practices around this?

Thanks!
Jared

On Fri, Jul 27, 2012 at 8:57 PM, Jared Cantwell <ja...@gmail.com>wrote:

> We are trying to remove the need for all admin intervention so that is one
> failure scenario that is interesting to us.
>
> Jared
>
>
> On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com> wrote:
>
> Yes, this entry will be deleted. I don't like this either - if a new
> follower reboots before added to the config it will not be able to boot up
> without manual help from an admin. That's why I'm considering maybe to
> remove the check that a participant must always initially be in its own
> config, but for now its there.
>
> Alex
>
> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <ja...@gmail.com>wrote:
>
>> Sorry for the confusion in terminology, I was unfamiliar with the exact
>> leader/follower semantics previously.
>>
>> So if all connected servers update their config file, does that mean that
>> non-voting followers who aren't part of the new ensemble will lose the
>> entry specific to them in their config file?  I can test this myself, but
>> getting an inside perspective is very helpful.
>>
>> Thanks again for the help!
>> Jared
>>
>>
>> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>
>> Yes, any number of followers which are not in the configuration can just
>> connect and listen in. This has always been the case, also in 3.4, I just
>> made use of this for the purpose of adding members during reconfiguration.
>> Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
>> where the leader actually counts the votes of anyone connected,
>> regardless of config membership :) This is fixed in ZK-107, so they are
>> really non-voting followers.
>>
>> >   I am assuming that's the case, and that it is a follower (and not
>> > participant) by virtue of not being in the official configuration
>> stored in
>> > zookeeper itself.
>>
>> Follower and participant types of servers is not something that was
>> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
>> Its just that the votes of participants that are not in the configuration
>> are not counted that's why we call them non-voting followers. BTW,
>> obviously a non-voting follower can not become leader (like ZK-1113 this
>> was also not enforced before ZK-107).
>>
>> > And a followup... does zookeeper only overwrite the dynamic
>> > configuration file for nodes that are voting participants?  Such that
>> if I
>> > started a follower and then left it running through some
>> > reconfigurations, its file would not get updated if it was never added
>> as
>> > part of those reconfigurations?
>>
>> No, as soon as it connects to the current leader, its dynamic config file
>> is overwritten with the current configuration as part of the
>> synchronization with the leader. Every time a new configuration is
>> committed, all connected servers (voting, non-voting, observers) will
>> update their dynamic config file, doesn't matter if they're in the config.
>>
>> Alex
>>
>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <jared.cantwell@gmail.com
>> > wrote:
>>
>>> So does just having the server started and pointing to the existing
>>> ensemble automatically make it a "non participating follower"?  In other
>>> words, there is no need to inform the existing nodes that this new node is
>>> joining as a follower?  And to extend that, there could be any number of
>>> followers that are simply listening in on the event stream?  I am assuming
>>> that's the case, and that it is a follower (and not participant) by virtue
>>> of not being in the official configuration stored in zookeeper itself.
>>>
>>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>
>>>> there are just two supported types - participant and observer.
>>>> (participant can act as either follower or leader).
>>>>
>>>> So you can either write participant or leave it unspecified (which
>>>> means participant by default). Also, since the ip is the same for all your
>>>> ports you don't have to write it twice.  All of these should work in the
>>>> same way:
>>>>
>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>> server.5=10.10.5.17:2182:2183:participant;2181<http://10.10.5.17:2181/>
>>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>>>
>>>>
>>>>
>>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <
>>>> jared.cantwell@gmail.com> wrote:
>>>>
>>>>> Thanks Alex for the response.  Our current lines in the configuration
>>>>> look like this:
>>>>>
>>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>>
>>>>> For the new servers is it ok for their entry to have "participant"?
>>>>>  Or should that be something different (e.g. "follower")?
>>>>>
>>>>> ~Jared
>>>>>
>>>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>>
>>>>>> Hi Jared,
>>>>>>
>>>>>> Thanks for experimenting with this feature.
>>>>>>
>>>>>> The idea is that new servers join as "non voting followers". Which
>>>>>> means that they act as normal followers but the leader ignores their votes
>>>>>> since they are not part of the current configuration. The leader only
>>>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>>>> of the new config is ready before the new config can be
>>>>>> committed/activated). Defining them as observers is not a good idea, for
>>>>>> example in your scenario if they were observers they wouldn't be able to
>>>>>> participate in the reconfiguration protocol (which is similar to the
>>>>>> protocol for committing any other operation in which observers don't
>>>>>> participate) and since we don't have a quorum of followers in the new
>>>>>> config that can ack, reconfiguration would throw an exception (of
>>>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>>>> Of course if you intend them to be observers in the new config you
>>>>>> can define them as observers since their votes are not needed during
>>>>>> reconfig anyway.
>>>>>>
>>>>>> You're right, the new servers must be able to connect to the old
>>>>>> quorum. At minimum, their file should contain the current leader, but
>>>>>> you can also copy the current configuration file to the new members
>>>>>> if you wish.
>>>>>>
>>>>>> In addition, you should add a line for the member itself, so that
>>>>>> server F appears in F's config file (Its not important that the other new
>>>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>>>> server itself is in the configuration its started with, otherwise its not
>>>>>> going to run. This check has always been there, but I'm thinking of
>>>>>> possibly changing it in the future.
>>>>>>
>>>>>> As soon as F connects to the leader, its config file will be
>>>>>> overwritten with the current config file as part of the synchronization
>>>>>> process.
>>>>>>
>>>>>> Alex
>>>>>>
>>>>>>
>>>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>>>> jared.cantwell@gmail.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>>>> have a
>>>>>>> question.  If I have a current set of servers in my ensemble
>>>>>>> {A,B,C,D,E}
>>>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>>>>>> dynamic config file on servers F,G,H be configured on startup?
>>>>>>>  Should they
>>>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>>>> ensembles?
>>>>>>>  It seems like these new servers need to  know about the old quorum,
>>>>>>> but
>>>>>>> since they aren't part of it yet its not clear to me how they should
>>>>>>> be
>>>>>>> configured.  Should there be an intermediate configuration with F,G,
>>>>>>> and H
>>>>>>> as simply Observers?
>>>>>>>
>>>>>>> I can't find much documentation on this so I want to make sure I
>>>>>>> understand
>>>>>>> things correctly.
>>>>>>>
>>>>>>> Thanks!
>>>>>>> ~Jared
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

We are trying to remove the need for all admin intervention so that is one failure scenario that is interesting to us. 

Jared

On Jul 27, 2012, at 7:42 PM, Alexander Shraer <sh...@gmail.com> wrote:

> Yes, this entry will be deleted. I don't like this either - if a new follower reboots before added to the config it will not be able to boot up without manual help from an admin. That's why I'm considering maybe to remove the check that a participant must always initially be in its own config, but for now its there.
> 
> Alex
> 
> On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <ja...@gmail.com> wrote:
> Sorry for the confusion in terminology, I was unfamiliar with the exact leader/follower semantics previously. 
> 
> So if all connected servers update their config file, does that mean that non-voting followers who aren't part of the new ensemble will lose the entry specific to them in their config file?  I can test this myself, but getting an inside perspective is very helpful. 
> 
> Thanks again for the help!
> Jared
> 
> 
> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
> 
>> Yes, any number of followers which are not in the configuration can just connect and listen in. This has always been the case, also in 3.4, I just made use of this for the purpose of adding members during reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113
>> where the leader actually counts the votes of anyone connected, regardless of config membership :) This is fixed in ZK-107, so they are really non-voting followers. 
>> 
>> >   I am assuming that's the case, and that it is a follower (and not 
>> > participant) by virtue of not being in the official configuration stored in 
>> > zookeeper itself. 
>> 
>> Follower and participant types of servers is not something that was defined in ZK-107. In ZooKeeper every follower/leader is a "participant". Its just that the votes of participants that are not in the configuration are not counted that's why we call them non-voting followers. BTW, obviously a non-voting follower can not become leader (like ZK-1113 this was also not enforced before ZK-107).
>> 
>> > And a followup... does zookeeper only overwrite the dynamic 
>> > configuration file for nodes that are voting participants?  Such that if I 
>> > started a follower and then left it running through some 
>> > reconfigurations, its file would not get updated if it was never added as 
>> > part of those reconfigurations?
>> 
>> No, as soon as it connects to the current leader, its dynamic config file is overwritten with the current configuration as part of the synchronization with the leader. Every time a new configuration is committed, all connected servers (voting, non-voting, observers) will update their dynamic config file, doesn't matter if they're in the config.
>> 
>> Alex
>> 
>> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <ja...@gmail.com> wrote:
>> So does just having the server started and pointing to the existing ensemble automatically make it a "non participating follower"?  In other words, there is no need to inform the existing nodes that this new node is joining as a follower?  And to extend that, there could be any number of followers that are simply listening in on the event stream?  I am assuming that's the case, and that it is a follower (and not participant) by virtue of not being in the official configuration stored in zookeeper itself.
>> 
>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com> wrote:
>> there are just two supported types - participant and observer.
>> (participant can act as either follower or leader).
>> 
>> So you can either write participant or leave it unspecified (which means participant by default). Also, since the ip is the same for all your ports you don't have to write it twice.  All of these should work in the same way:
>> 
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183:participant;2181
>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183;2181
>> 
>> 
>> 
>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <ja...@gmail.com> wrote:
>> Thanks Alex for the response.  Our current lines in the configuration look like this:
>> 
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> 
>> For the new servers is it ok for their entry to have "participant"?  Or should that be something different (e.g. "follower")?
>> 
>> ~Jared
>> 
>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com> wrote:
>> Hi Jared,
>> 
>> Thanks for experimenting with this feature. 
>> 
>> The idea is that new servers join as "non voting followers". Which means that they act as normal followers but the leader ignores their votes since they are not part of the current configuration. The leader only counts their votes during the reconfiguration itself (to make sure a quorum of the new config is ready before the new config can be committed/activated). Defining them as observers is not a good idea, for example in your scenario if they were observers they wouldn't be able to participate in the reconfiguration protocol (which is similar to the protocol for committing any other operation in which observers don't participate) and since we don't have a quorum of followers in the new config that can ack, reconfiguration would throw an exception (of KeeperException.NEWCONFIGNOQUORUM type). 
>> Of course if you intend them to be observers in the new config you can define them as observers since their votes are not needed during reconfig anyway.
>> 
>> You're right, the new servers must be able to connect to the old quorum. At minimum, their file should contain the current leader, but 
>> you can also copy the current configuration file to the new members if you wish. 
>> 
>> In addition, you should add a line for the member itself, so that server F appears in F's config file (Its not important that the other new servers appear in F's file, but it won't hurt either, so you can do a union of old and new if you wish). The constructor of QuorumPeer checks that the server itself is in the configuration its started with, otherwise its not going to run. This check has always been there, but I'm thinking of possibly changing it in the future.
>> 
>> As soon as F connects to the leader, its config file will be overwritten with the current config file as part of the synchronization process. 
>> 
>> Alex
>> 
>> 
>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <ja...@gmail.com> wrote:
>> Hi,
>> 
>> We are testing integration with 3.5.0 and dynamic membership and I have a
>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>> dynamic config file on servers F,G,H be configured on startup?  Should they
>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>  It seems like these new servers need to  know about the old quorum, but
>> since they aren't part of it yet its not clear to me how they should be
>> configured.  Should there be an intermediate configuration with F,G, and H
>> as simply Observers?
>> 
>> I can't find much documentation on this so I want to make sure I understand
>> things correctly.
>> 
>> Thanks!
>> ~Jared
>> 
>> 
>> 
>> 
>> 
> 

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

Yes, this entry will be deleted. I don't like this either - if a new
follower reboots before added to the config it will not be able to boot up
without manual help from an admin. That's why I'm considering maybe to
remove the check that a participant must always initially be in its own
config, but for now its there.

Alex

On Fri, Jul 27, 2012 at 6:34 PM, Jared Cantwell <ja...@gmail.com>wrote:

> Sorry for the confusion in terminology, I was unfamiliar with the exact
> leader/follower semantics previously.
>
> So if all connected servers update their config file, does that mean that
> non-voting followers who aren't part of the new ensemble will lose the
> entry specific to them in their config file?  I can test this myself, but
> getting an inside perspective is very helpful.
>
> Thanks again for the help!
> Jared
>
>
> On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>
> Yes, any number of followers which are not in the configuration can just
> connect and listen in. This has always been the case, also in 3.4, I just
> made use of this for the purpose of adding members during reconfiguration.
> Moreover, in 3.4 there this bug ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
> where the leader actually counts the votes of anyone connected, regardless
> of config membership :) This is fixed in ZK-107, so they are really
> non-voting followers.
>
> >   I am assuming that's the case, and that it is a follower (and not
> > participant) by virtue of not being in the official configuration stored
> in
> > zookeeper itself.
>
> Follower and participant types of servers is not something that was
> defined in ZK-107. In ZooKeeper every follower/leader is a "participant".
> Its just that the votes of participants that are not in the configuration
> are not counted that's why we call them non-voting followers. BTW,
> obviously a non-voting follower can not become leader (like ZK-1113 this
> was also not enforced before ZK-107).
>
> > And a followup... does zookeeper only overwrite the dynamic
> > configuration file for nodes that are voting participants?  Such that if
> I
> > started a follower and then left it running through some
> > reconfigurations, its file would not get updated if it was never added
> as
> > part of those reconfigurations?
>
> No, as soon as it connects to the current leader, its dynamic config file
> is overwritten with the current configuration as part of the
> synchronization with the leader. Every time a new configuration is
> committed, all connected servers (voting, non-voting, observers) will
> update their dynamic config file, doesn't matter if they're in the config.
>
> Alex
>
> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <ja...@gmail.com>wrote:
>
>> So does just having the server started and pointing to the existing
>> ensemble automatically make it a "non participating follower"?  In other
>> words, there is no need to inform the existing nodes that this new node is
>> joining as a follower?  And to extend that, there could be any number of
>> followers that are simply listening in on the event stream?  I am assuming
>> that's the case, and that it is a follower (and not participant) by virtue
>> of not being in the official configuration stored in zookeeper itself.
>>
>> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>
>>> there are just two supported types - participant and observer.
>>> (participant can act as either follower or leader).
>>>
>>> So you can either write participant or leave it unspecified (which means
>>> participant by default). Also, since the ip is the same for all your ports
>>> you don't have to write it twice.  All of these should work in the same way:
>>>
>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>> server.5=10.10.5.17:2182:2183:participant;2181 <http://10.10.5.17:2181/>
>>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>>
>>>
>>>
>>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <
>>> jared.cantwell@gmail.com> wrote:
>>>
>>>> Thanks Alex for the response.  Our current lines in the configuration
>>>> look like this:
>>>>
>>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>>
>>>> For the new servers is it ok for their entry to have "participant"?  Or
>>>> should that be something different (e.g. "follower")?
>>>>
>>>> ~Jared
>>>>
>>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>>
>>>>> Hi Jared,
>>>>>
>>>>> Thanks for experimenting with this feature.
>>>>>
>>>>> The idea is that new servers join as "non voting followers". Which
>>>>> means that they act as normal followers but the leader ignores their votes
>>>>> since they are not part of the current configuration. The leader only
>>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>>> of the new config is ready before the new config can be
>>>>> committed/activated). Defining them as observers is not a good idea, for
>>>>> example in your scenario if they were observers they wouldn't be able to
>>>>> participate in the reconfiguration protocol (which is similar to the
>>>>> protocol for committing any other operation in which observers don't
>>>>> participate) and since we don't have a quorum of followers in the new
>>>>> config that can ack, reconfiguration would throw an exception (of
>>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>>> Of course if you intend them to be observers in the new config you can
>>>>> define them as observers since their votes are not needed during reconfig
>>>>> anyway.
>>>>>
>>>>> You're right, the new servers must be able to connect to the old
>>>>> quorum. At minimum, their file should contain the current leader, but
>>>>> you can also copy the current configuration file to the new members if
>>>>> you wish.
>>>>>
>>>>> In addition, you should add a line for the member itself, so that
>>>>> server F appears in F's config file (Its not important that the other new
>>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>>> server itself is in the configuration its started with, otherwise its not
>>>>> going to run. This check has always been there, but I'm thinking of
>>>>> possibly changing it in the future.
>>>>>
>>>>> As soon as F connects to the leader, its config file will be
>>>>> overwritten with the current config file as part of the synchronization
>>>>> process.
>>>>>
>>>>> Alex
>>>>>
>>>>>
>>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>>> jared.cantwell@gmail.com> wrote:
>>>>>
>>>>>> Hi,
>>>>>>
>>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>>> have a
>>>>>> question.  If I have a current set of servers in my ensemble
>>>>>> {A,B,C,D,E}
>>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>>>>> dynamic config file on servers F,G,H be configured on startup?
>>>>>>  Should they
>>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>>> ensembles?
>>>>>>  It seems like these new servers need to  know about the old quorum,
>>>>>> but
>>>>>> since they aren't part of it yet its not clear to me how they should
>>>>>> be
>>>>>> configured.  Should there be an intermediate configuration with F,G,
>>>>>> and H
>>>>>> as simply Observers?
>>>>>>
>>>>>> I can't find much documentation on this so I want to make sure I
>>>>>> understand
>>>>>> things correctly.
>>>>>>
>>>>>> Thanks!
>>>>>> ~Jared
>>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

Sorry for the confusion in terminology, I was unfamiliar with the exact leader/follower semantics previously. 

So if all connected servers update their config file, does that mean that non-voting followers who aren't part of the new ensemble will lose the entry specific to them in their config file?  I can test this myself, but getting an inside perspective is very helpful. 

Thanks again for the help!
Jared

On Jul 27, 2012, at 6:55 PM, Alexander Shraer <sh...@gmail.com> wrote:

> Yes, any number of followers which are not in the configuration can just connect and listen in. This has always been the case, also in 3.4, I just made use of this for the purpose of adding members during reconfiguration. Moreover, in 3.4 there this bug ZOOKEEPER-1113
> where the leader actually counts the votes of anyone connected, regardless of config membership :) This is fixed in ZK-107, so they are really non-voting followers. 
> 
> >   I am assuming that's the case, and that it is a follower (and not 
> > participant) by virtue of not being in the official configuration stored in 
> > zookeeper itself. 
> 
> Follower and participant types of servers is not something that was defined in ZK-107. In ZooKeeper every follower/leader is a "participant". Its just that the votes of participants that are not in the configuration are not counted that's why we call them non-voting followers. BTW, obviously a non-voting follower can not become leader (like ZK-1113 this was also not enforced before ZK-107).
> 
> > And a followup... does zookeeper only overwrite the dynamic 
> > configuration file for nodes that are voting participants?  Such that if I 
> > started a follower and then left it running through some 
> > reconfigurations, its file would not get updated if it was never added as 
> > part of those reconfigurations?
> 
> No, as soon as it connects to the current leader, its dynamic config file is overwritten with the current configuration as part of the synchronization with the leader. Every time a new configuration is committed, all connected servers (voting, non-voting, observers) will update their dynamic config file, doesn't matter if they're in the config.
> 
> Alex
> 
> On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <ja...@gmail.com> wrote:
> So does just having the server started and pointing to the existing ensemble automatically make it a "non participating follower"?  In other words, there is no need to inform the existing nodes that this new node is joining as a follower?  And to extend that, there could be any number of followers that are simply listening in on the event stream?  I am assuming that's the case, and that it is a follower (and not participant) by virtue of not being in the official configuration stored in zookeeper itself.
> 
> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com> wrote:
> there are just two supported types - participant and observer.
> (participant can act as either follower or leader).
> 
> So you can either write participant or leave it unspecified (which means participant by default). Also, since the ip is the same for all your ports you don't have to write it twice.  All of these should work in the same way:
> 
> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
> server.5=10.10.5.17:2182:2183:participant;2181
> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
> server.5=10.10.5.17:2182:2183;2181
> 
> 
> 
> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <ja...@gmail.com> wrote:
> Thanks Alex for the response.  Our current lines in the configuration look like this:
> 
> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
> 
> For the new servers is it ok for their entry to have "participant"?  Or should that be something different (e.g. "follower")?
> 
> ~Jared
> 
> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com> wrote:
> Hi Jared,
> 
> Thanks for experimenting with this feature. 
> 
> The idea is that new servers join as "non voting followers". Which means that they act as normal followers but the leader ignores their votes since they are not part of the current configuration. The leader only counts their votes during the reconfiguration itself (to make sure a quorum of the new config is ready before the new config can be committed/activated). Defining them as observers is not a good idea, for example in your scenario if they were observers they wouldn't be able to participate in the reconfiguration protocol (which is similar to the protocol for committing any other operation in which observers don't participate) and since we don't have a quorum of followers in the new config that can ack, reconfiguration would throw an exception (of KeeperException.NEWCONFIGNOQUORUM type). 
> Of course if you intend them to be observers in the new config you can define them as observers since their votes are not needed during reconfig anyway.
> 
> You're right, the new servers must be able to connect to the old quorum. At minimum, their file should contain the current leader, but 
> you can also copy the current configuration file to the new members if you wish. 
> 
> In addition, you should add a line for the member itself, so that server F appears in F's config file (Its not important that the other new servers appear in F's file, but it won't hurt either, so you can do a union of old and new if you wish). The constructor of QuorumPeer checks that the server itself is in the configuration its started with, otherwise its not going to run. This check has always been there, but I'm thinking of possibly changing it in the future.
> 
> As soon as F connects to the leader, its config file will be overwritten with the current config file as part of the synchronization process. 
> 
> Alex
> 
> 
> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <ja...@gmail.com> wrote:
> Hi,
> 
> We are testing integration with 3.5.0 and dynamic membership and I have a
> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
> dynamic config file on servers F,G,H be configured on startup?  Should they
> have the old ensemble, the new ensemble, or the union of both ensembles?
>  It seems like these new servers need to  know about the old quorum, but
> since they aren't part of it yet its not clear to me how they should be
> configured.  Should there be an intermediate configuration with F,G, and H
> as simply Observers?
> 
> I can't find much documentation on this so I want to make sure I understand
> things correctly.
> 
> Thanks!
> ~Jared
> 
> 
> 
> 
> 

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

Yes, any number of followers which are not in the configuration can just
connect and listen in. This has always been the case, also in 3.4, I just
made use of this for the purpose of adding members during reconfiguration.
Moreover, in 3.4 there this bug
ZOOKEEPER-1113<https://issues.apache.org/jira/browse/ZOOKEEPER-1113>
where the leader actually counts the votes of anyone connected, regardless
of config membership :) This is fixed in ZK-107, so they are really
non-voting followers.

>   I am assuming that's the case, and that it is a follower (and not
> participant) by virtue of not being in the official configuration stored
in
> zookeeper itself.

Follower and participant types of servers is not something that was defined
in ZK-107. In ZooKeeper every follower/leader is a "participant". Its just
that the votes of participants that are not in the configuration are not
counted that's why we call them non-voting followers. BTW, obviously a
non-voting follower can not become leader (like ZK-1113 this was also not
enforced before ZK-107).

> And a followup... does zookeeper only overwrite the dynamic
> configuration file for nodes that are voting participants?  Such that if
I
> started a follower and then left it running through some
> reconfigurations, its file would not get updated if it was never added as
> part of those reconfigurations?

No, as soon as it connects to the current leader, its dynamic config file
is overwritten with the current configuration as part of the
synchronization with the leader. Every time a new configuration is
committed, all connected servers (voting, non-voting, observers) will
update their dynamic config file, doesn't matter if they're in the config.

Alex

On Fri, Jul 27, 2012 at 5:35 PM, Jared Cantwell <ja...@gmail.com>wrote:

> So does just having the server started and pointing to the existing
> ensemble automatically make it a "non participating follower"?  In other
> words, there is no need to inform the existing nodes that this new node is
> joining as a follower?  And to extend that, there could be any number of
> followers that are simply listening in on the event stream?  I am assuming
> that's the case, and that it is a follower (and not participant) by virtue
> of not being in the official configuration stored in zookeeper itself.
>
> On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com>wrote:
>
>> there are just two supported types - participant and observer.
>> (participant can act as either follower or leader).
>>
>> So you can either write participant or leave it unspecified (which means
>> participant by default). Also, since the ip is the same for all your ports
>> you don't have to write it twice.  All of these should work in the same way:
>>
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183:participant;2181 <http://10.10.5.17:2181/>
>> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
>> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>>
>>
>>
>> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <jared.cantwell@gmail.com
>> > wrote:
>>
>>> Thanks Alex for the response.  Our current lines in the configuration
>>> look like this:
>>>
>>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>>
>>> For the new servers is it ok for their entry to have "participant"?  Or
>>> should that be something different (e.g. "follower")?
>>>
>>> ~Jared
>>>
>>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>>
>>>> Hi Jared,
>>>>
>>>> Thanks for experimenting with this feature.
>>>>
>>>> The idea is that new servers join as "non voting followers". Which
>>>> means that they act as normal followers but the leader ignores their votes
>>>> since they are not part of the current configuration. The leader only
>>>> counts their votes during the reconfiguration itself (to make sure a quorum
>>>> of the new config is ready before the new config can be
>>>> committed/activated). Defining them as observers is not a good idea, for
>>>> example in your scenario if they were observers they wouldn't be able to
>>>> participate in the reconfiguration protocol (which is similar to the
>>>> protocol for committing any other operation in which observers don't
>>>> participate) and since we don't have a quorum of followers in the new
>>>> config that can ack, reconfiguration would throw an exception (of
>>>> KeeperException.NEWCONFIGNOQUORUM type).
>>>> Of course if you intend them to be observers in the new config you can
>>>> define them as observers since their votes are not needed during reconfig
>>>> anyway.
>>>>
>>>> You're right, the new servers must be able to connect to the old
>>>> quorum. At minimum, their file should contain the current leader, but
>>>> you can also copy the current configuration file to the new members if
>>>> you wish.
>>>>
>>>> In addition, you should add a line for the member itself, so that
>>>> server F appears in F's config file (Its not important that the other new
>>>> servers appear in F's file, but it won't hurt either, so you can do a union
>>>> of old and new if you wish). The constructor of QuorumPeer checks that the
>>>> server itself is in the configuration its started with, otherwise its not
>>>> going to run. This check has always been there, but I'm thinking of
>>>> possibly changing it in the future.
>>>>
>>>> As soon as F connects to the leader, its config file will be
>>>> overwritten with the current config file as part of the synchronization
>>>> process.
>>>>
>>>> Alex
>>>>
>>>>
>>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>>> jared.cantwell@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> We are testing integration with 3.5.0 and dynamic membership and I
>>>>> have a
>>>>> question.  If I have a current set of servers in my ensemble
>>>>> {A,B,C,D,E}
>>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>>>> dynamic config file on servers F,G,H be configured on startup?  Should
>>>>> they
>>>>> have the old ensemble, the new ensemble, or the union of both
>>>>> ensembles?
>>>>>  It seems like these new servers need to  know about the old quorum,
>>>>> but
>>>>> since they aren't part of it yet its not clear to me how they should be
>>>>> configured.  Should there be an intermediate configuration with F,G,
>>>>> and H
>>>>> as simply Observers?
>>>>>
>>>>> I can't find much documentation on this so I want to make sure I
>>>>> understand
>>>>> things correctly.
>>>>>
>>>>> Thanks!
>>>>> ~Jared
>>>>>
>>>>
>>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

So does just having the server started and pointing to the existing
ensemble automatically make it a "non participating follower"?  In other
words, there is no need to inform the existing nodes that this new node is
joining as a follower?  And to extend that, there could be any number of
followers that are simply listening in on the event stream?  I am assuming
that's the case, and that it is a follower (and not participant) by virtue
of not being in the official configuration stored in zookeeper itself.

On Fri, Jul 27, 2012 at 6:29 PM, Alexander Shraer <sh...@gmail.com> wrote:

> there are just two supported types - participant and observer.
> (participant can act as either follower or leader).
>
> So you can either write participant or leave it unspecified (which means
> participant by default). Also, since the ip is the same for all your ports
> you don't have to write it twice.  All of these should work in the same way:
>
> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
> server.5=10.10.5.17:2182:2183:participant;2181 <http://10.10.5.17:2181/>
> server.5=10.10.5.17:2182:2183;10.10.5.17:2181
> server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>
>
>
>
> On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <ja...@gmail.com>wrote:
>
>> Thanks Alex for the response.  Our current lines in the configuration
>> look like this:
>>
>> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>>
>> For the new servers is it ok for their entry to have "participant"?  Or
>> should that be something different (e.g. "follower")?
>>
>> ~Jared
>>
>> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>>
>>> Hi Jared,
>>>
>>> Thanks for experimenting with this feature.
>>>
>>> The idea is that new servers join as "non voting followers". Which means
>>> that they act as normal followers but the leader ignores their votes since
>>> they are not part of the current configuration. The leader only counts
>>> their votes during the reconfiguration itself (to make sure a quorum of the
>>> new config is ready before the new config can be committed/activated).
>>> Defining them as observers is not a good idea, for example in your scenario
>>> if they were observers they wouldn't be able to participate in the
>>> reconfiguration protocol (which is similar to the protocol for committing
>>> any other operation in which observers don't participate) and since we
>>> don't have a quorum of followers in the new config that can ack,
>>> reconfiguration would throw an exception (of
>>> KeeperException.NEWCONFIGNOQUORUM type).
>>> Of course if you intend them to be observers in the new config you can
>>> define them as observers since their votes are not needed during reconfig
>>> anyway.
>>>
>>> You're right, the new servers must be able to connect to the old quorum.
>>> At minimum, their file should contain the current leader, but
>>> you can also copy the current configuration file to the new members if
>>> you wish.
>>>
>>> In addition, you should add a line for the member itself, so that server
>>> F appears in F's config file (Its not important that the other new servers
>>> appear in F's file, but it won't hurt either, so you can do a union of old
>>> and new if you wish). The constructor of QuorumPeer checks that the server
>>> itself is in the configuration its started with, otherwise its not going to
>>> run. This check has always been there, but I'm thinking of possibly
>>> changing it in the future.
>>>
>>> As soon as F connects to the leader, its config file will be overwritten
>>> with the current config file as part of the synchronization process.
>>>
>>> Alex
>>>
>>>
>>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>>> jared.cantwell@gmail.com> wrote:
>>>
>>>> Hi,
>>>>
>>>> We are testing integration with 3.5.0 and dynamic membership and I have
>>>> a
>>>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>>> dynamic config file on servers F,G,H be configured on startup?  Should
>>>> they
>>>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>>>  It seems like these new servers need to  know about the old quorum, but
>>>> since they aren't part of it yet its not clear to me how they should be
>>>> configured.  Should there be an intermediate configuration with F,G,
>>>> and H
>>>> as simply Observers?
>>>>
>>>> I can't find much documentation on this so I want to make sure I
>>>> understand
>>>> things correctly.
>>>>
>>>> Thanks!
>>>> ~Jared
>>>>
>>>
>>>
>>
>

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

there are just two supported types - participant and observer.
(participant can act as either follower or leader).

So you can either write participant or leave it unspecified (which means
participant by default). Also, since the ip is the same for all your ports
you don't have to write it twice.  All of these should work in the same way:

server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
server.5=10.10.5.17:2182:2183:participant;2181 <http://10.10.5.17:2181/>
server.5=10.10.5.17:2182:2183;10.10.5.17:2181
server.5=10.10.5.17:2182:2183;2181 <http://10.10.5.17:2181/>



On Fri, Jul 27, 2012 at 5:25 PM, Jared Cantwell <ja...@gmail.com>wrote:

> Thanks Alex for the response.  Our current lines in the configuration look
> like this:
>
> server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181
>
> For the new servers is it ok for their entry to have "participant"?  Or
> should that be something different (e.g. "follower")?
>
> ~Jared
>
> On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com>wrote:
>
>> Hi Jared,
>>
>> Thanks for experimenting with this feature.
>>
>> The idea is that new servers join as "non voting followers". Which means
>> that they act as normal followers but the leader ignores their votes since
>> they are not part of the current configuration. The leader only counts
>> their votes during the reconfiguration itself (to make sure a quorum of the
>> new config is ready before the new config can be committed/activated).
>> Defining them as observers is not a good idea, for example in your scenario
>> if they were observers they wouldn't be able to participate in the
>> reconfiguration protocol (which is similar to the protocol for committing
>> any other operation in which observers don't participate) and since we
>> don't have a quorum of followers in the new config that can ack,
>> reconfiguration would throw an exception (of
>> KeeperException.NEWCONFIGNOQUORUM type).
>> Of course if you intend them to be observers in the new config you can
>> define them as observers since their votes are not needed during reconfig
>> anyway.
>>
>> You're right, the new servers must be able to connect to the old quorum.
>> At minimum, their file should contain the current leader, but
>> you can also copy the current configuration file to the new members if
>> you wish.
>>
>> In addition, you should add a line for the member itself, so that server
>> F appears in F's config file (Its not important that the other new servers
>> appear in F's file, but it won't hurt either, so you can do a union of old
>> and new if you wish). The constructor of QuorumPeer checks that the server
>> itself is in the configuration its started with, otherwise its not going to
>> run. This check has always been there, but I'm thinking of possibly
>> changing it in the future.
>>
>> As soon as F connects to the leader, its config file will be overwritten
>> with the current config file as part of the synchronization process.
>>
>> Alex
>>
>>
>> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <
>> jared.cantwell@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> We are testing integration with 3.5.0 and dynamic membership and I have a
>>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>>> dynamic config file on servers F,G,H be configured on startup?  Should
>>> they
>>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>>  It seems like these new servers need to  know about the old quorum, but
>>> since they aren't part of it yet its not clear to me how they should be
>>> configured.  Should there be an intermediate configuration with F,G, and
>>> H
>>> as simply Observers?
>>>
>>> I can't find much documentation on this so I want to make sure I
>>> understand
>>> things correctly.
>>>
>>> Thanks!
>>> ~Jared
>>>
>>
>>
>

Re: Dynamic reconfiguration

[Jared Cantwell <ja...@gmail.com>]

Thanks Alex for the response.  Our current lines in the configuration look
like this:

server.5=10.10.5.17:2182:2183:participant;10.10.5.17:2181

For the new servers is it ok for their entry to have "participant"?  Or
should that be something different (e.g. "follower")?

~Jared

On Fri, Jul 27, 2012 at 6:20 PM, Alexander Shraer <sh...@gmail.com> wrote:

> Hi Jared,
>
> Thanks for experimenting with this feature.
>
> The idea is that new servers join as "non voting followers". Which means
> that they act as normal followers but the leader ignores their votes since
> they are not part of the current configuration. The leader only counts
> their votes during the reconfiguration itself (to make sure a quorum of the
> new config is ready before the new config can be committed/activated).
> Defining them as observers is not a good idea, for example in your scenario
> if they were observers they wouldn't be able to participate in the
> reconfiguration protocol (which is similar to the protocol for committing
> any other operation in which observers don't participate) and since we
> don't have a quorum of followers in the new config that can ack,
> reconfiguration would throw an exception (of
> KeeperException.NEWCONFIGNOQUORUM type).
> Of course if you intend them to be observers in the new config you can
> define them as observers since their votes are not needed during reconfig
> anyway.
>
> You're right, the new servers must be able to connect to the old quorum.
> At minimum, their file should contain the current leader, but
> you can also copy the current configuration file to the new members if you
> wish.
>
> In addition, you should add a line for the member itself, so that server F
> appears in F's config file (Its not important that the other new servers
> appear in F's file, but it won't hurt either, so you can do a union of old
> and new if you wish). The constructor of QuorumPeer checks that the server
> itself is in the configuration its started with, otherwise its not going to
> run. This check has always been there, but I'm thinking of possibly
> changing it in the future.
>
> As soon as F connects to the leader, its config file will be overwritten
> with the current config file as part of the synchronization process.
>
> Alex
>
>
> On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell <jared.cantwell@gmail.com
> > wrote:
>
>> Hi,
>>
>> We are testing integration with 3.5.0 and dynamic membership and I have a
>> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
>> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
>> dynamic config file on servers F,G,H be configured on startup?  Should
>> they
>> have the old ensemble, the new ensemble, or the union of both ensembles?
>>  It seems like these new servers need to  know about the old quorum, but
>> since they aren't part of it yet its not clear to me how they should be
>> configured.  Should there be an intermediate configuration with F,G, and H
>> as simply Observers?
>>
>> I can't find much documentation on this so I want to make sure I
>> understand
>> things correctly.
>>
>> Thanks!
>> ~Jared
>>
>
>

Re: Dynamic reconfiguration

[Alexander Shraer <sh...@gmail.com>]

Hi Jared,

Thanks for experimenting with this feature.

The idea is that new servers join as "non voting followers". Which means
that they act as normal followers but the leader ignores their votes since
they are not part of the current configuration. The leader only counts
their votes during the reconfiguration itself (to make sure a quorum of the
new config is ready before the new config can be committed/activated).
Defining them as observers is not a good idea, for example in your scenario
if they were observers they wouldn't be able to participate in the
reconfiguration protocol (which is similar to the protocol for committing
any other operation in which observers don't participate) and since we
don't have a quorum of followers in the new config that can ack,
reconfiguration would throw an exception (of
KeeperException.NEWCONFIGNOQUORUM type).
Of course if you intend them to be observers in the new config you can
define them as observers since their votes are not needed during reconfig
anyway.

You're right, the new servers must be able to connect to the old quorum. At
minimum, their file should contain the current leader, but
you can also copy the current configuration file to the new members if you
wish.

In addition, you should add a line for the member itself, so that server F
appears in F's config file (Its not important that the other new servers
appear in F's file, but it won't hurt either, so you can do a union of old
and new if you wish). The constructor of QuorumPeer checks that the server
itself is in the configuration its started with, otherwise its not going to
run. This check has always been there, but I'm thinking of possibly
changing it in the future.

As soon as F connects to the leader, its config file will be overwritten
with the current config file as part of the synchronization process.

Alex


On Fri, Jul 27, 2012 at 10:06 AM, Jared Cantwell
<ja...@gmail.com>wrote:

> Hi,
>
> We are testing integration with 3.5.0 and dynamic membership and I have a
> question.  If I have a current set of servers in my ensemble {A,B,C,D,E}
> and I want to reconfigure the ensemble to {D,E,F,G,H}, how should the
> dynamic config file on servers F,G,H be configured on startup?  Should they
> have the old ensemble, the new ensemble, or the union of both ensembles?
>  It seems like these new servers need to  know about the old quorum, but
> since they aren't part of it yet its not clear to me how they should be
> configured.  Should there be an intermediate configuration with F,G, and H
> as simply Observers?
>
> I can't find much documentation on this so I want to make sure I understand
> things correctly.
>
> Thanks!
> ~Jared
>