You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Mohammad Shamma <mo...@gmail.com> on 2013/06/22 00:00:31 UTC
Zookeeper Configuration Sync
I have a use case where I dynamically grow a zookeeper ensemble on the same
fixed set of machines multiple times. In each iteration, the ensemble is
grown incrementally till it consists of "n" servers. I will refer to the
machines hosting the servers as zk-1, zk-2, ..., zk-n.
At the beginning of each iteration, I wipe out the zookeeper data
directories of zk-1 and zk-2, then statically configure the zookeeper
servers on both of them to form a 2-way ensemble. After that, I start
growing the ensemble incrementally by reconfiguring the zookeeper ensemble
to include zk-i, then clearing, configure and starting the zookeeper server
on zk-i (that is for i in range(2,n)).
I was not shutting down or cleaning up the previous ensemble zookeeper
servers at the end of each iteration. After initializing the 2-way ensemble
on zk-1 and zk-2, I observed that the servers from the old deployment were
contacting the servers of the new ensemble and triggering an ensemble
reconfiguration. A quick look at the code seems to suggest that this is
simply triggered by the virtue that the config version value of the old
deployment server is higher than that of that found on the new ensemble
servers. Can anyone confirm my understanding of this behaviour of zookeeper?
I also noticed that his reconfiguration holds true for n=3. For example
lets say zookeeper servers on zk-1 and zk-2 are freshly configured to form
a 2-way ensemble, and zk-3 contains a leftover server that was part of an
older 3-way ensemble (that included two obselete servers on zk-1 and zk-2).
To me it seems a bit counter intuitive for one server (on zk-3) to drive
the configuration of two other servers (zk1, zk2). The reason why it
seems counter intuitive is that the majority of the servers in the ensemble
agree on a different config version. Can somebody explain how zookeeper
handles this situation?
One final note, it would be really useful if a zookeeper ensemble would
have a unique identifier that could be set in the "zoo.cfg" file. Whenever
servers communicate witch each other, they would verify that they are
talking to peers of the same ensemble before commencing with further
actions. Does that sound like a reasonable request?
Thanks,
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Alexander Shraer <sh...@gmail.com>.
Right - otherwise it won't be able to initially connect to the ensemble.
On Jul 8, 2013 4:01 PM, "Mohammad Shamma" <mo...@gmail.com> wrote:
> Hmm, but a seed zookeeper config file is required to bootstrap the joiner
> anyways, right?
>
> --
> Mohammad Shamma
>
Re: Zookeeper Configuration Sync
Posted by Mohammad Shamma <mo...@gmail.com>.
Hmm, but a seed zookeeper config file is required to bootstrap the joiner
anyways, right?
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Alexander Shraer <sh...@gmail.com>.
It should be automatically updated for the joiner too.
On Jul 8, 2013 3:50 PM, "Mohammad Shamma" <mo...@gmail.com> wrote:
> Sure, the files are updated automatically for all servers except the
> joiner. I was referring to generating the config file for the joiner.
>
>
> On Mon, Jul 8, 2013 at 3:39 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > That's not necessary - during the reconfiguration the config files are
> > updated automatically.
> > On Jul 8, 2013 3:08 PM, "Mohammad Shamma" <mo...@gmail.com>
> > wrote:
> >
> > > Thanks for the documentation draft reference.
> > >
> > > I have been using the configuration information read using the zk
> client
> > > (post making the reconfig call) as a basis for generating the config
> file
> > > of the new zookeeper server.
> > >
> > >
> > > On Mon, Jul 8, 2013 at 2:48 PM, Alexander Shraer <sh...@gmail.com>
> > > wrote:
> > >
> > > > To your question, you can initialize the joiners config with a list
> > > > including the current servers and the new one. Or current leader and
> > new
> > > > server - works too but more fragile. If two servers join they
> shouldn't
> > > > have each other in their initial configs. See ZK-1660 for user
> manual.
> > > > On Jul 8, 2013 1:27 PM, "Mohammad Shamma" <mo...@gmail.com>
> > > > wrote:
> > > >
> > > > > Sorry for the late reply,
> > > > >
> > > > >
> > > > > On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <
> > shralex@gmail.com
> > > > > >wrote:
> > > > >
> > > > > > Hi Mohammad,
> > > > > >
> > > > > > +1 for the unique ensemble identifier request. We actually
> > discussed
> > > > > > this a long time ago but somehow never got to do this.
> > > > > > Can you open a JIRA for this ?
> > > > > >
> > > > >
> > > > > I will do that.
> > > > >
> > > > >
> > > > > >
> > > > > > Suppose that server A has the latest log but only talks with
> > server B
> > > > > > during leader election (C is down or slow). A doesn't know
> whether
> > > > > > or not the latest operations in its log are committed (in this
> > case C
> > > > > > would have them, but A doesn't know if this is the case). So to
> be
> > > > > > safe
> > > > > > everything in A's log gets committed in this case.
> > > > >
> > > > >
> > > > > > We took the approach that a reconfiguration is treated similarly
> to
> > > > > > normal data updates. When a server has the most up-to-date log
> and
> > > > > > talks with a majority during leader election, it will be elected
> > > > > > leader and commit its log to the other servers. It won't truncate
> > its
> > > > > > log even
> > > > > > if its clear that some operations were not committed. This is
> true
> > > for
> > > > > > normal updates as well as for reconfigurations.
> > > > > >
> > > > > > BTW, I'm not sure why you are shutting down servers or clearing
> the
> > > > > > data during reconfigurations, or why you're manually changing
> > config
> > > > > > files.
> > > > >
> > > > >
> > > > > The reason I am shutting down and clearing the data of the "to be
> > > added"
> > > > > servers is to delete their history as the intention here is to make
> > > them
> > > > > join a new fresh deployment.
> > > > >
> > > > > You can add servers to the ensemble by invoking the reconfig
> command
> > > > > > and this will make all the necessary changes to the config files,
> > > > > > including specifying the right config version.
> > > > > >
> > > > >
> > > > > If that is the case, what goes into the zookeeper config file of a
> > new
> > > > > zookeeper server that is supposed to join an existing ensemble?
> > > > >
> > > > >
> > > > > >
> > > > > > Alex
> > > > > >
> > > > > >
> > > > > > On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> > > > > > <mo...@gmail.com> wrote:
> > > > > > > I have a use case where I dynamically grow a zookeeper ensemble
> > on
> > > > the
> > > > > > same
> > > > > > > fixed set of machines multiple times. In each iteration, the
> > > ensemble
> > > > > is
> > > > > > > grown incrementally till it consists of "n" servers. I will
> refer
> > > to
> > > > > the
> > > > > > > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> > > > > > >
> > > > > > > At the beginning of each iteration, I wipe out the zookeeper
> data
> > > > > > > directories of zk-1 and zk-2, then statically configure the
> > > zookeeper
> > > > > > > servers on both of them to form a 2-way ensemble. After that, I
> > > start
> > > > > > > growing the ensemble incrementally by reconfiguring the
> zookeeper
> > > > > > ensemble
> > > > > > > to include zk-i, then clearing, configure and starting the
> > > zookeeper
> > > > > > server
> > > > > > > on zk-i (that is for i in range(2,n)).
> > > > > > >
> > > > > > > I was not shutting down or cleaning up the previous ensemble
> > > > zookeeper
> > > > > > > servers at the end of each iteration. After initializing the
> > 2-way
> > > > > > ensemble
> > > > > > > on zk-1 and zk-2, I observed that the servers from the old
> > > deployment
> > > > > > were
> > > > > > > contacting the servers of the new ensemble and triggering an
> > > ensemble
> > > > > > > reconfiguration. A quick look at the code seems to suggest that
> > > this
> > > > is
> > > > > > > simply triggered by the virtue that the config version value of
> > the
> > > > old
> > > > > > > deployment server is higher than that of that found on the new
> > > > ensemble
> > > > > > > servers. Can anyone confirm my understanding of this behaviour
> of
> > > > > > zookeeper?
> > > > > > >
> > > > > > > I also noticed that his reconfiguration holds true for n=3. For
> > > > example
> > > > > > > lets say zookeeper servers on zk-1 and zk-2 are freshly
> > configured
> > > > to
> > > > > > form
> > > > > > > a 2-way ensemble, and zk-3 contains a leftover server that was
> > part
> > > > of
> > > > > an
> > > > > > > older 3-way ensemble (that included two obselete servers on
> zk-1
> > > and
> > > > > > zk-2).
> > > > > > > To me it seems a bit counter intuitive for one server (on zk-3)
> > to
> > > > > drive
> > > > > > > the configuration of two other servers (zk1, zk2). The reason
> why
> > > it
> > > > > > > seems counter intuitive is that the majority of the servers in
> > the
> > > > > > ensemble
> > > > > > > agree on a different config version. Can somebody explain how
> > > > zookeeper
> > > > > > > handles this situation?
> > > > > > >
> > > > > > > One final note, it would be really useful if a zookeeper
> ensemble
> > > > would
> > > > > > > have a unique identifier that could be set in the "zoo.cfg"
> file.
> > > > > > Whenever
> > > > > > > servers communicate witch each other, they would verify that
> they
> > > are
> > > > > > > talking to peers of the same ensemble before commencing with
> > > further
> > > > > > > actions. Does that sound like a reasonable request?
> > > > > > >
> > > > > > > Thanks,
> > > > > > >
> > > > > > > --
> > > > > > > Mohammad Shamma
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Mohammad Shamma
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Mohammad Shamma
> > >
> >
>
>
>
> --
> Mohammad Shamma
>
Re: Zookeeper Configuration Sync
Posted by Mohammad Shamma <mo...@gmail.com>.
Sure, the files are updated automatically for all servers except the
joiner. I was referring to generating the config file for the joiner.
On Mon, Jul 8, 2013 at 3:39 PM, Alexander Shraer <sh...@gmail.com> wrote:
> That's not necessary - during the reconfiguration the config files are
> updated automatically.
> On Jul 8, 2013 3:08 PM, "Mohammad Shamma" <mo...@gmail.com>
> wrote:
>
> > Thanks for the documentation draft reference.
> >
> > I have been using the configuration information read using the zk client
> > (post making the reconfig call) as a basis for generating the config file
> > of the new zookeeper server.
> >
> >
> > On Mon, Jul 8, 2013 at 2:48 PM, Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > To your question, you can initialize the joiners config with a list
> > > including the current servers and the new one. Or current leader and
> new
> > > server - works too but more fragile. If two servers join they shouldn't
> > > have each other in their initial configs. See ZK-1660 for user manual.
> > > On Jul 8, 2013 1:27 PM, "Mohammad Shamma" <mo...@gmail.com>
> > > wrote:
> > >
> > > > Sorry for the late reply,
> > > >
> > > >
> > > > On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <
> shralex@gmail.com
> > > > >wrote:
> > > >
> > > > > Hi Mohammad,
> > > > >
> > > > > +1 for the unique ensemble identifier request. We actually
> discussed
> > > > > this a long time ago but somehow never got to do this.
> > > > > Can you open a JIRA for this ?
> > > > >
> > > >
> > > > I will do that.
> > > >
> > > >
> > > > >
> > > > > Suppose that server A has the latest log but only talks with
> server B
> > > > > during leader election (C is down or slow). A doesn't know whether
> > > > > or not the latest operations in its log are committed (in this
> case C
> > > > > would have them, but A doesn't know if this is the case). So to be
> > > > > safe
> > > > > everything in A's log gets committed in this case.
> > > >
> > > >
> > > > > We took the approach that a reconfiguration is treated similarly to
> > > > > normal data updates. When a server has the most up-to-date log and
> > > > > talks with a majority during leader election, it will be elected
> > > > > leader and commit its log to the other servers. It won't truncate
> its
> > > > > log even
> > > > > if its clear that some operations were not committed. This is true
> > for
> > > > > normal updates as well as for reconfigurations.
> > > > >
> > > > > BTW, I'm not sure why you are shutting down servers or clearing the
> > > > > data during reconfigurations, or why you're manually changing
> config
> > > > > files.
> > > >
> > > >
> > > > The reason I am shutting down and clearing the data of the "to be
> > added"
> > > > servers is to delete their history as the intention here is to make
> > them
> > > > join a new fresh deployment.
> > > >
> > > > You can add servers to the ensemble by invoking the reconfig command
> > > > > and this will make all the necessary changes to the config files,
> > > > > including specifying the right config version.
> > > > >
> > > >
> > > > If that is the case, what goes into the zookeeper config file of a
> new
> > > > zookeeper server that is supposed to join an existing ensemble?
> > > >
> > > >
> > > > >
> > > > > Alex
> > > > >
> > > > >
> > > > > On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> > > > > <mo...@gmail.com> wrote:
> > > > > > I have a use case where I dynamically grow a zookeeper ensemble
> on
> > > the
> > > > > same
> > > > > > fixed set of machines multiple times. In each iteration, the
> > ensemble
> > > > is
> > > > > > grown incrementally till it consists of "n" servers. I will refer
> > to
> > > > the
> > > > > > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> > > > > >
> > > > > > At the beginning of each iteration, I wipe out the zookeeper data
> > > > > > directories of zk-1 and zk-2, then statically configure the
> > zookeeper
> > > > > > servers on both of them to form a 2-way ensemble. After that, I
> > start
> > > > > > growing the ensemble incrementally by reconfiguring the zookeeper
> > > > > ensemble
> > > > > > to include zk-i, then clearing, configure and starting the
> > zookeeper
> > > > > server
> > > > > > on zk-i (that is for i in range(2,n)).
> > > > > >
> > > > > > I was not shutting down or cleaning up the previous ensemble
> > > zookeeper
> > > > > > servers at the end of each iteration. After initializing the
> 2-way
> > > > > ensemble
> > > > > > on zk-1 and zk-2, I observed that the servers from the old
> > deployment
> > > > > were
> > > > > > contacting the servers of the new ensemble and triggering an
> > ensemble
> > > > > > reconfiguration. A quick look at the code seems to suggest that
> > this
> > > is
> > > > > > simply triggered by the virtue that the config version value of
> the
> > > old
> > > > > > deployment server is higher than that of that found on the new
> > > ensemble
> > > > > > servers. Can anyone confirm my understanding of this behaviour of
> > > > > zookeeper?
> > > > > >
> > > > > > I also noticed that his reconfiguration holds true for n=3. For
> > > example
> > > > > > lets say zookeeper servers on zk-1 and zk-2 are freshly
> configured
> > > to
> > > > > form
> > > > > > a 2-way ensemble, and zk-3 contains a leftover server that was
> part
> > > of
> > > > an
> > > > > > older 3-way ensemble (that included two obselete servers on zk-1
> > and
> > > > > zk-2).
> > > > > > To me it seems a bit counter intuitive for one server (on zk-3)
> to
> > > > drive
> > > > > > the configuration of two other servers (zk1, zk2). The reason why
> > it
> > > > > > seems counter intuitive is that the majority of the servers in
> the
> > > > > ensemble
> > > > > > agree on a different config version. Can somebody explain how
> > > zookeeper
> > > > > > handles this situation?
> > > > > >
> > > > > > One final note, it would be really useful if a zookeeper ensemble
> > > would
> > > > > > have a unique identifier that could be set in the "zoo.cfg" file.
> > > > > Whenever
> > > > > > servers communicate witch each other, they would verify that they
> > are
> > > > > > talking to peers of the same ensemble before commencing with
> > further
> > > > > > actions. Does that sound like a reasonable request?
> > > > > >
> > > > > > Thanks,
> > > > > >
> > > > > > --
> > > > > > Mohammad Shamma
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Mohammad Shamma
> > > >
> > >
> >
> >
> >
> > --
> > Mohammad Shamma
> >
>
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Alexander Shraer <sh...@gmail.com>.
That's not necessary - during the reconfiguration the config files are
updated automatically.
On Jul 8, 2013 3:08 PM, "Mohammad Shamma" <mo...@gmail.com> wrote:
> Thanks for the documentation draft reference.
>
> I have been using the configuration information read using the zk client
> (post making the reconfig call) as a basis for generating the config file
> of the new zookeeper server.
>
>
> On Mon, Jul 8, 2013 at 2:48 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > To your question, you can initialize the joiners config with a list
> > including the current servers and the new one. Or current leader and new
> > server - works too but more fragile. If two servers join they shouldn't
> > have each other in their initial configs. See ZK-1660 for user manual.
> > On Jul 8, 2013 1:27 PM, "Mohammad Shamma" <mo...@gmail.com>
> > wrote:
> >
> > > Sorry for the late reply,
> > >
> > >
> > > On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <shralex@gmail.com
> > > >wrote:
> > >
> > > > Hi Mohammad,
> > > >
> > > > +1 for the unique ensemble identifier request. We actually discussed
> > > > this a long time ago but somehow never got to do this.
> > > > Can you open a JIRA for this ?
> > > >
> > >
> > > I will do that.
> > >
> > >
> > > >
> > > > Suppose that server A has the latest log but only talks with server B
> > > > during leader election (C is down or slow). A doesn't know whether
> > > > or not the latest operations in its log are committed (in this case C
> > > > would have them, but A doesn't know if this is the case). So to be
> > > > safe
> > > > everything in A's log gets committed in this case.
> > >
> > >
> > > > We took the approach that a reconfiguration is treated similarly to
> > > > normal data updates. When a server has the most up-to-date log and
> > > > talks with a majority during leader election, it will be elected
> > > > leader and commit its log to the other servers. It won't truncate its
> > > > log even
> > > > if its clear that some operations were not committed. This is true
> for
> > > > normal updates as well as for reconfigurations.
> > > >
> > > > BTW, I'm not sure why you are shutting down servers or clearing the
> > > > data during reconfigurations, or why you're manually changing config
> > > > files.
> > >
> > >
> > > The reason I am shutting down and clearing the data of the "to be
> added"
> > > servers is to delete their history as the intention here is to make
> them
> > > join a new fresh deployment.
> > >
> > > You can add servers to the ensemble by invoking the reconfig command
> > > > and this will make all the necessary changes to the config files,
> > > > including specifying the right config version.
> > > >
> > >
> > > If that is the case, what goes into the zookeeper config file of a new
> > > zookeeper server that is supposed to join an existing ensemble?
> > >
> > >
> > > >
> > > > Alex
> > > >
> > > >
> > > > On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> > > > <mo...@gmail.com> wrote:
> > > > > I have a use case where I dynamically grow a zookeeper ensemble on
> > the
> > > > same
> > > > > fixed set of machines multiple times. In each iteration, the
> ensemble
> > > is
> > > > > grown incrementally till it consists of "n" servers. I will refer
> to
> > > the
> > > > > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> > > > >
> > > > > At the beginning of each iteration, I wipe out the zookeeper data
> > > > > directories of zk-1 and zk-2, then statically configure the
> zookeeper
> > > > > servers on both of them to form a 2-way ensemble. After that, I
> start
> > > > > growing the ensemble incrementally by reconfiguring the zookeeper
> > > > ensemble
> > > > > to include zk-i, then clearing, configure and starting the
> zookeeper
> > > > server
> > > > > on zk-i (that is for i in range(2,n)).
> > > > >
> > > > > I was not shutting down or cleaning up the previous ensemble
> > zookeeper
> > > > > servers at the end of each iteration. After initializing the 2-way
> > > > ensemble
> > > > > on zk-1 and zk-2, I observed that the servers from the old
> deployment
> > > > were
> > > > > contacting the servers of the new ensemble and triggering an
> ensemble
> > > > > reconfiguration. A quick look at the code seems to suggest that
> this
> > is
> > > > > simply triggered by the virtue that the config version value of the
> > old
> > > > > deployment server is higher than that of that found on the new
> > ensemble
> > > > > servers. Can anyone confirm my understanding of this behaviour of
> > > > zookeeper?
> > > > >
> > > > > I also noticed that his reconfiguration holds true for n=3. For
> > example
> > > > > lets say zookeeper servers on zk-1 and zk-2 are freshly configured
> > to
> > > > form
> > > > > a 2-way ensemble, and zk-3 contains a leftover server that was part
> > of
> > > an
> > > > > older 3-way ensemble (that included two obselete servers on zk-1
> and
> > > > zk-2).
> > > > > To me it seems a bit counter intuitive for one server (on zk-3) to
> > > drive
> > > > > the configuration of two other servers (zk1, zk2). The reason why
> it
> > > > > seems counter intuitive is that the majority of the servers in the
> > > > ensemble
> > > > > agree on a different config version. Can somebody explain how
> > zookeeper
> > > > > handles this situation?
> > > > >
> > > > > One final note, it would be really useful if a zookeeper ensemble
> > would
> > > > > have a unique identifier that could be set in the "zoo.cfg" file.
> > > > Whenever
> > > > > servers communicate witch each other, they would verify that they
> are
> > > > > talking to peers of the same ensemble before commencing with
> further
> > > > > actions. Does that sound like a reasonable request?
> > > > >
> > > > > Thanks,
> > > > >
> > > > > --
> > > > > Mohammad Shamma
> > > >
> > >
> > >
> > >
> > > --
> > > Mohammad Shamma
> > >
> >
>
>
>
> --
> Mohammad Shamma
>
Re: Zookeeper Configuration Sync
Posted by Mohammad Shamma <mo...@gmail.com>.
Thanks for the documentation draft reference.
I have been using the configuration information read using the zk client
(post making the reconfig call) as a basis for generating the config file
of the new zookeeper server.
On Mon, Jul 8, 2013 at 2:48 PM, Alexander Shraer <sh...@gmail.com> wrote:
> To your question, you can initialize the joiners config with a list
> including the current servers and the new one. Or current leader and new
> server - works too but more fragile. If two servers join they shouldn't
> have each other in their initial configs. See ZK-1660 for user manual.
> On Jul 8, 2013 1:27 PM, "Mohammad Shamma" <mo...@gmail.com>
> wrote:
>
> > Sorry for the late reply,
> >
> >
> > On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <shralex@gmail.com
> > >wrote:
> >
> > > Hi Mohammad,
> > >
> > > +1 for the unique ensemble identifier request. We actually discussed
> > > this a long time ago but somehow never got to do this.
> > > Can you open a JIRA for this ?
> > >
> >
> > I will do that.
> >
> >
> > >
> > > Suppose that server A has the latest log but only talks with server B
> > > during leader election (C is down or slow). A doesn't know whether
> > > or not the latest operations in its log are committed (in this case C
> > > would have them, but A doesn't know if this is the case). So to be
> > > safe
> > > everything in A's log gets committed in this case.
> >
> >
> > > We took the approach that a reconfiguration is treated similarly to
> > > normal data updates. When a server has the most up-to-date log and
> > > talks with a majority during leader election, it will be elected
> > > leader and commit its log to the other servers. It won't truncate its
> > > log even
> > > if its clear that some operations were not committed. This is true for
> > > normal updates as well as for reconfigurations.
> > >
> > > BTW, I'm not sure why you are shutting down servers or clearing the
> > > data during reconfigurations, or why you're manually changing config
> > > files.
> >
> >
> > The reason I am shutting down and clearing the data of the "to be added"
> > servers is to delete their history as the intention here is to make them
> > join a new fresh deployment.
> >
> > You can add servers to the ensemble by invoking the reconfig command
> > > and this will make all the necessary changes to the config files,
> > > including specifying the right config version.
> > >
> >
> > If that is the case, what goes into the zookeeper config file of a new
> > zookeeper server that is supposed to join an existing ensemble?
> >
> >
> > >
> > > Alex
> > >
> > >
> > > On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> > > <mo...@gmail.com> wrote:
> > > > I have a use case where I dynamically grow a zookeeper ensemble on
> the
> > > same
> > > > fixed set of machines multiple times. In each iteration, the ensemble
> > is
> > > > grown incrementally till it consists of "n" servers. I will refer to
> > the
> > > > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> > > >
> > > > At the beginning of each iteration, I wipe out the zookeeper data
> > > > directories of zk-1 and zk-2, then statically configure the zookeeper
> > > > servers on both of them to form a 2-way ensemble. After that, I start
> > > > growing the ensemble incrementally by reconfiguring the zookeeper
> > > ensemble
> > > > to include zk-i, then clearing, configure and starting the zookeeper
> > > server
> > > > on zk-i (that is for i in range(2,n)).
> > > >
> > > > I was not shutting down or cleaning up the previous ensemble
> zookeeper
> > > > servers at the end of each iteration. After initializing the 2-way
> > > ensemble
> > > > on zk-1 and zk-2, I observed that the servers from the old deployment
> > > were
> > > > contacting the servers of the new ensemble and triggering an ensemble
> > > > reconfiguration. A quick look at the code seems to suggest that this
> is
> > > > simply triggered by the virtue that the config version value of the
> old
> > > > deployment server is higher than that of that found on the new
> ensemble
> > > > servers. Can anyone confirm my understanding of this behaviour of
> > > zookeeper?
> > > >
> > > > I also noticed that his reconfiguration holds true for n=3. For
> example
> > > > lets say zookeeper servers on zk-1 and zk-2 are freshly configured
> to
> > > form
> > > > a 2-way ensemble, and zk-3 contains a leftover server that was part
> of
> > an
> > > > older 3-way ensemble (that included two obselete servers on zk-1 and
> > > zk-2).
> > > > To me it seems a bit counter intuitive for one server (on zk-3) to
> > drive
> > > > the configuration of two other servers (zk1, zk2). The reason why it
> > > > seems counter intuitive is that the majority of the servers in the
> > > ensemble
> > > > agree on a different config version. Can somebody explain how
> zookeeper
> > > > handles this situation?
> > > >
> > > > One final note, it would be really useful if a zookeeper ensemble
> would
> > > > have a unique identifier that could be set in the "zoo.cfg" file.
> > > Whenever
> > > > servers communicate witch each other, they would verify that they are
> > > > talking to peers of the same ensemble before commencing with further
> > > > actions. Does that sound like a reasonable request?
> > > >
> > > > Thanks,
> > > >
> > > > --
> > > > Mohammad Shamma
> > >
> >
> >
> >
> > --
> > Mohammad Shamma
> >
>
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Alexander Shraer <sh...@gmail.com>.
To your question, you can initialize the joiners config with a list
including the current servers and the new one. Or current leader and new
server - works too but more fragile. If two servers join they shouldn't
have each other in their initial configs. See ZK-1660 for user manual.
On Jul 8, 2013 1:27 PM, "Mohammad Shamma" <mo...@gmail.com> wrote:
> Sorry for the late reply,
>
>
> On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <shralex@gmail.com
> >wrote:
>
> > Hi Mohammad,
> >
> > +1 for the unique ensemble identifier request. We actually discussed
> > this a long time ago but somehow never got to do this.
> > Can you open a JIRA for this ?
> >
>
> I will do that.
>
>
> >
> > Suppose that server A has the latest log but only talks with server B
> > during leader election (C is down or slow). A doesn't know whether
> > or not the latest operations in its log are committed (in this case C
> > would have them, but A doesn't know if this is the case). So to be
> > safe
> > everything in A's log gets committed in this case.
>
>
> > We took the approach that a reconfiguration is treated similarly to
> > normal data updates. When a server has the most up-to-date log and
> > talks with a majority during leader election, it will be elected
> > leader and commit its log to the other servers. It won't truncate its
> > log even
> > if its clear that some operations were not committed. This is true for
> > normal updates as well as for reconfigurations.
> >
> > BTW, I'm not sure why you are shutting down servers or clearing the
> > data during reconfigurations, or why you're manually changing config
> > files.
>
>
> The reason I am shutting down and clearing the data of the "to be added"
> servers is to delete their history as the intention here is to make them
> join a new fresh deployment.
>
> You can add servers to the ensemble by invoking the reconfig command
> > and this will make all the necessary changes to the config files,
> > including specifying the right config version.
> >
>
> If that is the case, what goes into the zookeeper config file of a new
> zookeeper server that is supposed to join an existing ensemble?
>
>
> >
> > Alex
> >
> >
> > On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> > <mo...@gmail.com> wrote:
> > > I have a use case where I dynamically grow a zookeeper ensemble on the
> > same
> > > fixed set of machines multiple times. In each iteration, the ensemble
> is
> > > grown incrementally till it consists of "n" servers. I will refer to
> the
> > > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> > >
> > > At the beginning of each iteration, I wipe out the zookeeper data
> > > directories of zk-1 and zk-2, then statically configure the zookeeper
> > > servers on both of them to form a 2-way ensemble. After that, I start
> > > growing the ensemble incrementally by reconfiguring the zookeeper
> > ensemble
> > > to include zk-i, then clearing, configure and starting the zookeeper
> > server
> > > on zk-i (that is for i in range(2,n)).
> > >
> > > I was not shutting down or cleaning up the previous ensemble zookeeper
> > > servers at the end of each iteration. After initializing the 2-way
> > ensemble
> > > on zk-1 and zk-2, I observed that the servers from the old deployment
> > were
> > > contacting the servers of the new ensemble and triggering an ensemble
> > > reconfiguration. A quick look at the code seems to suggest that this is
> > > simply triggered by the virtue that the config version value of the old
> > > deployment server is higher than that of that found on the new ensemble
> > > servers. Can anyone confirm my understanding of this behaviour of
> > zookeeper?
> > >
> > > I also noticed that his reconfiguration holds true for n=3. For example
> > > lets say zookeeper servers on zk-1 and zk-2 are freshly configured to
> > form
> > > a 2-way ensemble, and zk-3 contains a leftover server that was part of
> an
> > > older 3-way ensemble (that included two obselete servers on zk-1 and
> > zk-2).
> > > To me it seems a bit counter intuitive for one server (on zk-3) to
> drive
> > > the configuration of two other servers (zk1, zk2). The reason why it
> > > seems counter intuitive is that the majority of the servers in the
> > ensemble
> > > agree on a different config version. Can somebody explain how zookeeper
> > > handles this situation?
> > >
> > > One final note, it would be really useful if a zookeeper ensemble would
> > > have a unique identifier that could be set in the "zoo.cfg" file.
> > Whenever
> > > servers communicate witch each other, they would verify that they are
> > > talking to peers of the same ensemble before commencing with further
> > > actions. Does that sound like a reasonable request?
> > >
> > > Thanks,
> > >
> > > --
> > > Mohammad Shamma
> >
>
>
>
> --
> Mohammad Shamma
>
Re: Zookeeper Configuration Sync
Posted by Mohammad Shamma <mo...@gmail.com>.
FYI, I filed a ticket for that
ZOOKEEPER-1723<https://issues.apache.org/jira/browse/ZOOKEEPER-1723>
.
On Mon, Jul 8, 2013 at 1:26 PM, Mohammad Shamma <mo...@gmail.com>wrote:
> Sorry for the late reply,
>
>
> On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <sh...@gmail.com>wrote:
>
>> Hi Mohammad,
>>
>> +1 for the unique ensemble identifier request. We actually discussed
>> this a long time ago but somehow never got to do this.
>> Can you open a JIRA for this ?
>>
>
> I will do that.
>
>
>>
>> Suppose that server A has the latest log but only talks with server B
>> during leader election (C is down or slow). A doesn't know whether
>> or not the latest operations in its log are committed (in this case C
>> would have them, but A doesn't know if this is the case). So to be
>> safe
>> everything in A's log gets committed in this case.
>
>
>> We took the approach that a reconfiguration is treated similarly to
>> normal data updates. When a server has the most up-to-date log and
>> talks with a majority during leader election, it will be elected
>> leader and commit its log to the other servers. It won't truncate its
>> log even
>> if its clear that some operations were not committed. This is true for
>> normal updates as well as for reconfigurations.
>>
>> BTW, I'm not sure why you are shutting down servers or clearing the
>> data during reconfigurations, or why you're manually changing config
>> files.
>
>
> The reason I am shutting down and clearing the data of the "to be added"
> servers is to delete their history as the intention here is to make them
> join a new fresh deployment.
>
> You can add servers to the ensemble by invoking the reconfig command
>> and this will make all the necessary changes to the config files,
>> including specifying the right config version.
>>
>
> If that is the case, what goes into the zookeeper config file of a new
> zookeeper server that is supposed to join an existing ensemble?
>
>
>>
>> Alex
>>
>>
>> On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
>> <mo...@gmail.com> wrote:
>> > I have a use case where I dynamically grow a zookeeper ensemble on the
>> same
>> > fixed set of machines multiple times. In each iteration, the ensemble is
>> > grown incrementally till it consists of "n" servers. I will refer to the
>> > machines hosting the servers as zk-1, zk-2, ..., zk-n.
>> >
>> > At the beginning of each iteration, I wipe out the zookeeper data
>> > directories of zk-1 and zk-2, then statically configure the zookeeper
>> > servers on both of them to form a 2-way ensemble. After that, I start
>> > growing the ensemble incrementally by reconfiguring the zookeeper
>> ensemble
>> > to include zk-i, then clearing, configure and starting the zookeeper
>> server
>> > on zk-i (that is for i in range(2,n)).
>> >
>> > I was not shutting down or cleaning up the previous ensemble zookeeper
>> > servers at the end of each iteration. After initializing the 2-way
>> ensemble
>> > on zk-1 and zk-2, I observed that the servers from the old deployment
>> were
>> > contacting the servers of the new ensemble and triggering an ensemble
>> > reconfiguration. A quick look at the code seems to suggest that this is
>> > simply triggered by the virtue that the config version value of the old
>> > deployment server is higher than that of that found on the new ensemble
>> > servers. Can anyone confirm my understanding of this behaviour of
>> zookeeper?
>> >
>> > I also noticed that his reconfiguration holds true for n=3. For example
>> > lets say zookeeper servers on zk-1 and zk-2 are freshly configured to
>> form
>> > a 2-way ensemble, and zk-3 contains a leftover server that was part of
>> an
>> > older 3-way ensemble (that included two obselete servers on zk-1 and
>> zk-2).
>> > To me it seems a bit counter intuitive for one server (on zk-3) to drive
>> > the configuration of two other servers (zk1, zk2). The reason why it
>> > seems counter intuitive is that the majority of the servers in the
>> ensemble
>> > agree on a different config version. Can somebody explain how zookeeper
>> > handles this situation?
>> >
>> > One final note, it would be really useful if a zookeeper ensemble would
>> > have a unique identifier that could be set in the "zoo.cfg" file.
>> Whenever
>> > servers communicate witch each other, they would verify that they are
>> > talking to peers of the same ensemble before commencing with further
>> > actions. Does that sound like a reasonable request?
>> >
>> > Thanks,
>> >
>> > --
>> > Mohammad Shamma
>>
>
>
>
> --
> Mohammad Shamma
>
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Mohammad Shamma <mo...@gmail.com>.
Sorry for the late reply,
On Fri, Jun 21, 2013 at 11:47 PM, Alexander Shraer <sh...@gmail.com>wrote:
> Hi Mohammad,
>
> +1 for the unique ensemble identifier request. We actually discussed
> this a long time ago but somehow never got to do this.
> Can you open a JIRA for this ?
>
I will do that.
>
> Suppose that server A has the latest log but only talks with server B
> during leader election (C is down or slow). A doesn't know whether
> or not the latest operations in its log are committed (in this case C
> would have them, but A doesn't know if this is the case). So to be
> safe
> everything in A's log gets committed in this case.
> We took the approach that a reconfiguration is treated similarly to
> normal data updates. When a server has the most up-to-date log and
> talks with a majority during leader election, it will be elected
> leader and commit its log to the other servers. It won't truncate its
> log even
> if its clear that some operations were not committed. This is true for
> normal updates as well as for reconfigurations.
>
> BTW, I'm not sure why you are shutting down servers or clearing the
> data during reconfigurations, or why you're manually changing config
> files.
The reason I am shutting down and clearing the data of the "to be added"
servers is to delete their history as the intention here is to make them
join a new fresh deployment.
You can add servers to the ensemble by invoking the reconfig command
> and this will make all the necessary changes to the config files,
> including specifying the right config version.
>
If that is the case, what goes into the zookeeper config file of a new
zookeeper server that is supposed to join an existing ensemble?
>
> Alex
>
>
> On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
> <mo...@gmail.com> wrote:
> > I have a use case where I dynamically grow a zookeeper ensemble on the
> same
> > fixed set of machines multiple times. In each iteration, the ensemble is
> > grown incrementally till it consists of "n" servers. I will refer to the
> > machines hosting the servers as zk-1, zk-2, ..., zk-n.
> >
> > At the beginning of each iteration, I wipe out the zookeeper data
> > directories of zk-1 and zk-2, then statically configure the zookeeper
> > servers on both of them to form a 2-way ensemble. After that, I start
> > growing the ensemble incrementally by reconfiguring the zookeeper
> ensemble
> > to include zk-i, then clearing, configure and starting the zookeeper
> server
> > on zk-i (that is for i in range(2,n)).
> >
> > I was not shutting down or cleaning up the previous ensemble zookeeper
> > servers at the end of each iteration. After initializing the 2-way
> ensemble
> > on zk-1 and zk-2, I observed that the servers from the old deployment
> were
> > contacting the servers of the new ensemble and triggering an ensemble
> > reconfiguration. A quick look at the code seems to suggest that this is
> > simply triggered by the virtue that the config version value of the old
> > deployment server is higher than that of that found on the new ensemble
> > servers. Can anyone confirm my understanding of this behaviour of
> zookeeper?
> >
> > I also noticed that his reconfiguration holds true for n=3. For example
> > lets say zookeeper servers on zk-1 and zk-2 are freshly configured to
> form
> > a 2-way ensemble, and zk-3 contains a leftover server that was part of an
> > older 3-way ensemble (that included two obselete servers on zk-1 and
> zk-2).
> > To me it seems a bit counter intuitive for one server (on zk-3) to drive
> > the configuration of two other servers (zk1, zk2). The reason why it
> > seems counter intuitive is that the majority of the servers in the
> ensemble
> > agree on a different config version. Can somebody explain how zookeeper
> > handles this situation?
> >
> > One final note, it would be really useful if a zookeeper ensemble would
> > have a unique identifier that could be set in the "zoo.cfg" file.
> Whenever
> > servers communicate witch each other, they would verify that they are
> > talking to peers of the same ensemble before commencing with further
> > actions. Does that sound like a reasonable request?
> >
> > Thanks,
> >
> > --
> > Mohammad Shamma
>
--
Mohammad Shamma
Re: Zookeeper Configuration Sync
Posted by Alexander Shraer <sh...@gmail.com>.
Hi Mohammad,
+1 for the unique ensemble identifier request. We actually discussed
this a long time ago but somehow never got to do this.
Can you open a JIRA for this ?
Suppose that server A has the latest log but only talks with server B
during leader election (C is down or slow). A doesn't know whether
or not the latest operations in its log are committed (in this case C
would have them, but A doesn't know if this is the case). So to be
safe
everything in A's log gets committed in this case.
We took the approach that a reconfiguration is treated similarly to
normal data updates. When a server has the most up-to-date log and
talks with a majority during leader election, it will be elected
leader and commit its log to the other servers. It won't truncate its
log even
if its clear that some operations were not committed. This is true for
normal updates as well as for reconfigurations.
BTW, I'm not sure why you are shutting down servers or clearing the
data during reconfigurations, or why you're manually changing config
files.
You can add servers to the ensemble by invoking the reconfig command
and this will make all the necessary changes to the config files,
including specifying the right config version.
Alex
On Fri, Jun 21, 2013 at 3:00 PM, Mohammad Shamma
<mo...@gmail.com> wrote:
> I have a use case where I dynamically grow a zookeeper ensemble on the same
> fixed set of machines multiple times. In each iteration, the ensemble is
> grown incrementally till it consists of "n" servers. I will refer to the
> machines hosting the servers as zk-1, zk-2, ..., zk-n.
>
> At the beginning of each iteration, I wipe out the zookeeper data
> directories of zk-1 and zk-2, then statically configure the zookeeper
> servers on both of them to form a 2-way ensemble. After that, I start
> growing the ensemble incrementally by reconfiguring the zookeeper ensemble
> to include zk-i, then clearing, configure and starting the zookeeper server
> on zk-i (that is for i in range(2,n)).
>
> I was not shutting down or cleaning up the previous ensemble zookeeper
> servers at the end of each iteration. After initializing the 2-way ensemble
> on zk-1 and zk-2, I observed that the servers from the old deployment were
> contacting the servers of the new ensemble and triggering an ensemble
> reconfiguration. A quick look at the code seems to suggest that this is
> simply triggered by the virtue that the config version value of the old
> deployment server is higher than that of that found on the new ensemble
> servers. Can anyone confirm my understanding of this behaviour of zookeeper?
>
> I also noticed that his reconfiguration holds true for n=3. For example
> lets say zookeeper servers on zk-1 and zk-2 are freshly configured to form
> a 2-way ensemble, and zk-3 contains a leftover server that was part of an
> older 3-way ensemble (that included two obselete servers on zk-1 and zk-2).
> To me it seems a bit counter intuitive for one server (on zk-3) to drive
> the configuration of two other servers (zk1, zk2). The reason why it
> seems counter intuitive is that the majority of the servers in the ensemble
> agree on a different config version. Can somebody explain how zookeeper
> handles this situation?
>
> One final note, it would be really useful if a zookeeper ensemble would
> have a unique identifier that could be set in the "zoo.cfg" file. Whenever
> servers communicate witch each other, they would verify that they are
> talking to peers of the same ensemble before commencing with further
> actions. Does that sound like a reasonable request?
>
> Thanks,
>
> --
> Mohammad Shamma