You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Guillermo Vega-Toro <gv...@us.ibm.com> on 2017/06/21 18:33:52 UTC
How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Hi,
I have a 3-node Zookeeper 3.5.3-beta ensemble. This was initially created
by un-taring the Zookeeper tarball into 3 machines, editing the zoo.cfg
file with the 3 server entries, and starting the servers. The ensemble
started, and a zoo.cfg.dynamic.100000000 file was created with the
server.x lines in all servers.
Now, I want to add 2 more nodes to grow to a 5-node ensemble. In Zookeeper
3.4.6, I would install the two new nodes, update the zoo.cfg files on all
servers to add the new nodes, and restart the servers one by one. But in
version 3.5.3-beta, this is not possible - any changes made to
zoo.cfg.dynamic.100000000 are ignored, even if reconfiguration is
disabled. If the dynamic config file is modified to add new servers and
the server is restarted, the dynamic config file gets overwritten with the
old 3-node configuration when the server starts.
So, it looks like in Zookeeper 3.5.x, the way to add nodes is to use the
new reconfig command on the CLI. But if the reconfiguration feature is
disabled for security reasons (as it is in 3.5.3-beta by default), then
adding nodes "the old way" by editing the config file and restarting the
servers is not possible either. Does this mean that in Zookeeper
3.5.3-beta, in order to grow the ensemble, it is necessary to enable
reconfiguration (by setting reconfigEnabled=true in zoo.cfg and restarting
all servers), then running the CLI to add the new nodes? Then after the
new nodes are added, edit zoo.cfg to disable reconfiguration on all
servers, and restart all servers again?
Thanks
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Alexander Shraer <sh...@gmail.com>.
> 2. Even with the reconfig CLI, if there is no quorum, it is not possible
> to re-configue the ensemble, so one has to fall back to modify the
> ensemble through modification of zoo.cfg and restart.
Just like any other update operation in ZK, reconfig isn't available when
you loose quorum.
In your scenario reconfiguring from 5 to 2 without talking with the
disconnected 3 servers may
cause data loss. But I can understand that sometimes this may be needed. I
think you should
be able to change the config files to do that if you wanted to, but it may
be tricky without restarting
both at the same time since we use configuration ids to understand which
config is more up-to-date
and if you restart just one server and remove that id, the other server may
push its own config to that
second server during leader election...
On Fri, Jun 23, 2017 at 9:12 AM, Michael Han <ha...@cloudera.com> wrote:
> Guillermo, thank you for reporting the issue and share you findings on the
> workaround.
>
> I filed https://issues.apache.org/jira/browse/ZOOKEEPER-2819 which should
> provide expected behavior - when reconfigEnabled=false you should be able
> to do rolling restarts the old ways, once that JIRA is fixed.
>
>
> On Fri, Jun 23, 2017 at 7:18 AM, Guillermo Vega-Toro <gv...@us.ibm.com>
> wrote:
>
> > Thanks all for looking at this.
> >
> > Here is what I've found to make ensemble config changes work with
> > 3.5.3-beta and reconfigEnabled=false:
> >
> > 1. All servers must be stopped.
> > 2. In one server, make the desired changes to zoo.cfg (group, server.x,
> > weight.x), and delete the dynamicConfigFile property.
> > 3. Start the one server above. You will notice that when the server
> > starts, no dynamic file is created, as if the config is "on hold".
> > 4. In another server, make the same changes to zoo.cfg, and start the
> > server. If there is no quorum, no dynamic file will be created.
> > 5. Repeat step 4 for all servers
> >
> > Once a quorum is reached, the proposed config changes are applied, and a
> > dynamic config file will appear on the servers. If you run a 'config'
> > command on the zkCLI, the desired configuration will show.
> >
> > Also, once quorum is reached, further ensemble changes to zoo.cfg
> followed
> > by restart of a single server are ignored. It is necessary to stop all
> > servers and do the steps above to make any changes to the ensemble.
> >
> > Thanks,
> >
> > Alexander Shraer <sh...@gmail.com> wrote on 06/23/2017 01:20:47 AM:
> >
> > > From: Alexander Shraer <sh...@gmail.com>
> > > To: user@zookeeper.apache.org
> > > Date: 06/23/2017 01:21 AM
> > > Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> > > with reconfigEnabled=false
> > >
> > > I'm not sure it's necessary for backward compatibility since rolling
> > > restarts for config changes are not really an api the system provides.
> > >
> > > I'd think the ACL control and admin only API should be sufficient for
> > > security and would prefer to get rid of the flag. But if you must have
> > it,
> > > we have to prevent both in memory config updates (most important) and
> > > config file updates if reconfig is disabled. This sounds like a small
> > > change in quorumpeer, but perhaps I'm forgetting something.
> > >
> > > Cheers
> > > Alex
> > >
> > >
> > > On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com>
> wrote:
> > >
> > > > Hi Alex, thanks for clarification!
> > > >
> > > > It makes sense to me that users should use reconfig instead of
> rolling
> > > > upgrade moving forward. The only concern is backward compatibility
> now
> > we
> > > > drop the old rolling upgrade support: since 3.5.x needs to be
> backward
> > > > compatible with 3.4.x [1], I think we probably need support rolling
> > upgrade
> > > > for 3.5 branch.
> > > >
> > > > As for this flag - I believe it's there and set to false because
> > reconfig
> > > > is a security sensitive feature and for such features user has to opt
> > in
> > > > explicitly. Our security team here also has similar recommendations
> > when I
> > > > talked with them about what this feature could do. There are also
> some
> > > > discussions around this flag / why it's there in ZOOKEEPER-2014.
> > > >
> > > > [1]
> > > >
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <
> shralex@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi Michael,
> > > > >
> > > > > The described behavior is the intended one - in 3.5 configuration
> is
> > part
> > > > > of the synced state and is updated
> > > > > when the server syncs with the leader. The only rolling upgrade I
> > tested
> > > > > was to upgrade the software version
> > > > > of the servers - this should still work. But I didn't try to
> support
> > > > > rolling upgrade for upgrading the configuration,
> > > > > since this should be done through reconfig.
> > > > >
> > > > > I'm still not sure what's the purpose of this flag btw. Why would
> > someone
> > > > > want to do rolling restarts which are prone
> > > > > to inconsistencies and data loss, when they can use reconfig ?
> > > > >
> > > > > Alex
> > > > >
> > > > >
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com>
> > wrote:
> > > > >
> > > > > > reconfigEnabled only disables reconfig command when
> > > > > reconfigEnabled=false;
> > > > > > it does not disable the feature by mute all code paths of the
> > reconfig
> > > > > > feature introduced in ZOOKEEPER-107. So regardless of the value
> of
> > > > > > reconfigEnabled,
> > > > > > 3.5.x ZK will create static config file and dynamic config file
> in
> > any
> > > > > > cases.
> > > > > >
> > > > > > This might create a problem for users who want to do rolling
> > upgrade
> > > > the
> > > > > > old way - because now the critical config information is not
> > stored in
> > > > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not
> > work
> > > > > > because a reconfig needs to go through quorum processors. I think
> > this
> > > > is
> > > > > > the problem described in the thread.
> > > > > >
> > > > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > > > anything
> > > > > > mentioned in ZOOKEEPER-107 about this. Currently I think the
> > answer is
> > > > > no,
> > > > > > which means for 3.5.x the only way to change membership of
> cluster
> > is
> > > > > > through reconfig. Could you confirm this conclusion? If that is
> > the
> > > > case
> > > > > we
> > > > > > need patch the reconfigEnabled so it completely disable all code
> > path
> > > > of
> > > > > > the reconfig feature to leave the static zoo.cfg intact.
> > > > > >
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer
> > <sh...@gmail.com>
> > > > > > wrote:
> > > > > >
> > > > > > > This sounds like a bug in the implementation of
> reconfigEnabled.
> > > > > > > Could you please open a JIRA with the description you provided
> ?
> > > > > > >
> > > > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > > > exactly
> > > > > > > to perform the changes you're trying to make, in a simple and
> > correct
> > > > > > way.
> > > > > > >
> > > > > > > Thanks,
> > > > > > > Alex
> > > > > > >
> > > > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > > > gvegator@us.ibm.com>
> > > > > > > wrote:
> > > > > > >
> > > > > > > > I'm still unable to make configuration changes when
> > > > > > reconfigEnabled=false
> > > > > > > > by updating zoo.cfg and restarting the servers.
> > > > > > > >
> > > > > > > > For example, I want to change the weight of one of my
> servers.
> > I
> > > > edit
> > > > > > > > zoo.cfg on the server I want to change, and specify the
> group,
> > > > > > server.x,
> > > > > > > > and weight.x properties for all servers. I also remove the
> > > > > > > > dynamicConfigFile property and delete the dynamic config
> file.
> > I
> > > > then
> > > > > > > > restart the server. As soon as the server starts, the dynamic
> > > > config
> > > > > > file
> > > > > > > > re-appears, and it has the last configuration, as if the
> > changes I
> > > > > made
> > > > > > > in
> > > > > > > > zoo.cfg were ignored. The dynamic configuration file on the
> > other
> > > > > > servers
> > > > > > > > also stays the same.
> > > > > > > >
> > > > > > > > What would be the correct way to achieve this (change a
> > server's
> > > > > > weight,
> > > > > > > > or role) when reconfigEnabled=false and the CLI reconfig
> > command
> > > > > cannot
> > > > > > > be
> > > > > > > > used?
> > > > > > > >
> > > > > > > > Thanks
> > > > > > > >
> > > > > > > >
> > > > > > >
> > > > > >
> > > > > >
> > > > > >
> > > > > > --
> > > > > > Cheers
> > > > > > Michael.
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> >
> >
> >
>
>
> --
> Cheers
> Michael.
>
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
Guillermo, thank you for reporting the issue and share you findings on the
workaround.
I filed https://issues.apache.org/jira/browse/ZOOKEEPER-2819 which should
provide expected behavior - when reconfigEnabled=false you should be able
to do rolling restarts the old ways, once that JIRA is fixed.
On Fri, Jun 23, 2017 at 7:18 AM, Guillermo Vega-Toro <gv...@us.ibm.com>
wrote:
> Thanks all for looking at this.
>
> Here is what I've found to make ensemble config changes work with
> 3.5.3-beta and reconfigEnabled=false:
>
> 1. All servers must be stopped.
> 2. In one server, make the desired changes to zoo.cfg (group, server.x,
> weight.x), and delete the dynamicConfigFile property.
> 3. Start the one server above. You will notice that when the server
> starts, no dynamic file is created, as if the config is "on hold".
> 4. In another server, make the same changes to zoo.cfg, and start the
> server. If there is no quorum, no dynamic file will be created.
> 5. Repeat step 4 for all servers
>
> Once a quorum is reached, the proposed config changes are applied, and a
> dynamic config file will appear on the servers. If you run a 'config'
> command on the zkCLI, the desired configuration will show.
>
> Also, once quorum is reached, further ensemble changes to zoo.cfg followed
> by restart of a single server are ignored. It is necessary to stop all
> servers and do the steps above to make any changes to the ensemble.
>
> Thanks,
>
> Alexander Shraer <sh...@gmail.com> wrote on 06/23/2017 01:20:47 AM:
>
> > From: Alexander Shraer <sh...@gmail.com>
> > To: user@zookeeper.apache.org
> > Date: 06/23/2017 01:21 AM
> > Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> > with reconfigEnabled=false
> >
> > I'm not sure it's necessary for backward compatibility since rolling
> > restarts for config changes are not really an api the system provides.
> >
> > I'd think the ACL control and admin only API should be sufficient for
> > security and would prefer to get rid of the flag. But if you must have
> it,
> > we have to prevent both in memory config updates (most important) and
> > config file updates if reconfig is disabled. This sounds like a small
> > change in quorumpeer, but perhaps I'm forgetting something.
> >
> > Cheers
> > Alex
> >
> >
> > On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com> wrote:
> >
> > > Hi Alex, thanks for clarification!
> > >
> > > It makes sense to me that users should use reconfig instead of rolling
> > > upgrade moving forward. The only concern is backward compatibility now
> we
> > > drop the old rolling upgrade support: since 3.5.x needs to be backward
> > > compatible with 3.4.x [1], I think we probably need support rolling
> upgrade
> > > for 3.5 branch.
> > >
> > > As for this flag - I believe it's there and set to false because
> reconfig
> > > is a security sensitive feature and for such features user has to opt
> in
> > > explicitly. Our security team here also has similar recommendations
> when I
> > > talked with them about what this feature could do. There are also some
> > > discussions around this flag / why it's there in ZOOKEEPER-2014.
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <sh...@gmail.com>
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > The described behavior is the intended one - in 3.5 configuration is
> part
> > > > of the synced state and is updated
> > > > when the server syncs with the leader. The only rolling upgrade I
> tested
> > > > was to upgrade the software version
> > > > of the servers - this should still work. But I didn't try to support
> > > > rolling upgrade for upgrading the configuration,
> > > > since this should be done through reconfig.
> > > >
> > > > I'm still not sure what's the purpose of this flag btw. Why would
> someone
> > > > want to do rolling restarts which are prone
> > > > to inconsistencies and data loss, when they can use reconfig ?
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com>
> wrote:
> > > >
> > > > > reconfigEnabled only disables reconfig command when
> > > > reconfigEnabled=false;
> > > > > it does not disable the feature by mute all code paths of the
> reconfig
> > > > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > > > reconfigEnabled,
> > > > > 3.5.x ZK will create static config file and dynamic config file in
> any
> > > > > cases.
> > > > >
> > > > > This might create a problem for users who want to do rolling
> upgrade
> > > the
> > > > > old way - because now the critical config information is not
> stored in
> > > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not
> work
> > > > > because a reconfig needs to go through quorum processors. I think
> this
> > > is
> > > > > the problem described in the thread.
> > > > >
> > > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > > anything
> > > > > mentioned in ZOOKEEPER-107 about this. Currently I think the
> answer is
> > > > no,
> > > > > which means for 3.5.x the only way to change membership of cluster
> is
> > > > > through reconfig. Could you confirm this conclusion? If that is
> the
> > > case
> > > > we
> > > > > need patch the reconfigEnabled so it completely disable all code
> path
> > > of
> > > > > the reconfig feature to leave the static zoo.cfg intact.
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer
> <sh...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > > > Could you please open a JIRA with the description you provided ?
> > > > > >
> > > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > > exactly
> > > > > > to perform the changes you're trying to make, in a simple and
> correct
> > > > > way.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > > gvegator@us.ibm.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm still unable to make configuration changes when
> > > > > reconfigEnabled=false
> > > > > > > by updating zoo.cfg and restarting the servers.
> > > > > > >
> > > > > > > For example, I want to change the weight of one of my servers.
> I
> > > edit
> > > > > > > zoo.cfg on the server I want to change, and specify the group,
> > > > > server.x,
> > > > > > > and weight.x properties for all servers. I also remove the
> > > > > > > dynamicConfigFile property and delete the dynamic config file.
> I
> > > then
> > > > > > > restart the server. As soon as the server starts, the dynamic
> > > config
> > > > > file
> > > > > > > re-appears, and it has the last configuration, as if the
> changes I
> > > > made
> > > > > > in
> > > > > > > zoo.cfg were ignored. The dynamic configuration file on the
> other
> > > > > servers
> > > > > > > also stays the same.
> > > > > > >
> > > > > > > What would be the correct way to achieve this (change a
> server's
> > > > > weight,
> > > > > > > or role) when reconfigEnabled=false and the CLI reconfig
> command
> > > > cannot
> > > > > > be
> > > > > > > used?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers
> > > > > Michael.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
>
>
>
--
Cheers
Michael.
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with
reconfigEnabled=false
Posted by Guillermo Vega-Toro <gv...@us.ibm.com>.
Thanks all for looking at this.
Here is what I've found to make ensemble config changes work with
3.5.3-beta and reconfigEnabled=false:
1. All servers must be stopped.
2. In one server, make the desired changes to zoo.cfg (group, server.x,
weight.x), and delete the dynamicConfigFile property.
3. Start the one server above. You will notice that when the server
starts, no dynamic file is created, as if the config is "on hold".
4. In another server, make the same changes to zoo.cfg, and start the
server. If there is no quorum, no dynamic file will be created.
5. Repeat step 4 for all servers
Once a quorum is reached, the proposed config changes are applied, and a
dynamic config file will appear on the servers. If you run a 'config'
command on the zkCLI, the desired configuration will show.
Also, once quorum is reached, further ensemble changes to zoo.cfg followed
by restart of a single server are ignored. It is necessary to stop all
servers and do the steps above to make any changes to the ensemble.
Thanks,
Alexander Shraer <sh...@gmail.com> wrote on 06/23/2017 01:20:47 AM:
> From: Alexander Shraer <sh...@gmail.com>
> To: user@zookeeper.apache.org
> Date: 06/23/2017 01:21 AM
> Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> with reconfigEnabled=false
>
> I'm not sure it's necessary for backward compatibility since rolling
> restarts for config changes are not really an api the system provides.
>
> I'd think the ACL control and admin only API should be sufficient for
> security and would prefer to get rid of the flag. But if you must have
it,
> we have to prevent both in memory config updates (most important) and
> config file updates if reconfig is disabled. This sounds like a small
> change in quorumpeer, but perhaps I'm forgetting something.
>
> Cheers
> Alex
>
>
> On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com> wrote:
>
> > Hi Alex, thanks for clarification!
> >
> > It makes sense to me that users should use reconfig instead of rolling
> > upgrade moving forward. The only concern is backward compatibility now
we
> > drop the old rolling upgrade support: since 3.5.x needs to be backward
> > compatible with 3.4.x [1], I think we probably need support rolling
upgrade
> > for 3.5 branch.
> >
> > As for this flag - I believe it's there and set to false because
reconfig
> > is a security sensitive feature and for such features user has to opt
in
> > explicitly. Our security team here also has similar recommendations
when I
> > talked with them about what this feature could do. There are also some
> > discussions around this flag / why it's there in ZOOKEEPER-2014.
> >
> > [1]
> >
https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> >
> >
> > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > Hi Michael,
> > >
> > > The described behavior is the intended one - in 3.5 configuration is
part
> > > of the synced state and is updated
> > > when the server syncs with the leader. The only rolling upgrade I
tested
> > > was to upgrade the software version
> > > of the servers - this should still work. But I didn't try to support
> > > rolling upgrade for upgrading the configuration,
> > > since this should be done through reconfig.
> > >
> > > I'm still not sure what's the purpose of this flag btw. Why would
someone
> > > want to do rolling restarts which are prone
> > > to inconsistencies and data loss, when they can use reconfig ?
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com>
wrote:
> > >
> > > > reconfigEnabled only disables reconfig command when
> > > reconfigEnabled=false;
> > > > it does not disable the feature by mute all code paths of the
reconfig
> > > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > > reconfigEnabled,
> > > > 3.5.x ZK will create static config file and dynamic config file in
any
> > > > cases.
> > > >
> > > > This might create a problem for users who want to do rolling
upgrade
> > the
> > > > old way - because now the critical config information is not
stored in
> > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not
work
> > > > because a reconfig needs to go through quorum processors. I think
this
> > is
> > > > the problem described in the thread.
> > > >
> > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > anything
> > > > mentioned in ZOOKEEPER-107 about this. Currently I think the
answer is
> > > no,
> > > > which means for 3.5.x the only way to change membership of cluster
is
> > > > through reconfig. Could you confirm this conclusion? If that is
the
> > case
> > > we
> > > > need patch the reconfigEnabled so it completely disable all code
path
> > of
> > > > the reconfig feature to leave the static zoo.cfg intact.
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer
<sh...@gmail.com>
> > > > wrote:
> > > >
> > > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > > Could you please open a JIRA with the description you provided ?
> > > > >
> > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > exactly
> > > > > to perform the changes you're trying to make, in a simple and
correct
> > > > way.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > gvegator@us.ibm.com>
> > > > > wrote:
> > > > >
> > > > > > I'm still unable to make configuration changes when
> > > > reconfigEnabled=false
> > > > > > by updating zoo.cfg and restarting the servers.
> > > > > >
> > > > > > For example, I want to change the weight of one of my servers.
I
> > edit
> > > > > > zoo.cfg on the server I want to change, and specify the group,
> > > > server.x,
> > > > > > and weight.x properties for all servers. I also remove the
> > > > > > dynamicConfigFile property and delete the dynamic config file.
I
> > then
> > > > > > restart the server. As soon as the server starts, the dynamic
> > config
> > > > file
> > > > > > re-appears, and it has the last configuration, as if the
changes I
> > > made
> > > > > in
> > > > > > zoo.cfg were ignored. The dynamic configuration file on the
other
> > > > servers
> > > > > > also stays the same.
> > > > > >
> > > > > > What would be the correct way to achieve this (change a
server's
> > > > weight,
> > > > > > or role) when reconfigEnabled=false and the CLI reconfig
command
> > > cannot
> > > > > be
> > > > > > used?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
On Thu, Jun 22, 2017 at 11:20 PM, Alexander Shraer <sh...@gmail.com>
wrote:
> I'm not sure it's necessary for backward compatibility since rolling
> restarts for config changes are not really an api the system provides.
>
Yeah, it is not part of API in a strict sense but there are tools /
management software build on top of rolling restarts. Also the principal of
least surprise to users - especially given that we did not explicitly
mention in the doc that rolling restarts is deprecated on 3.5.x (the doc
only mentioned rolling restart is 'not needed' anymore :). It seems
reasonable to provide an option to at least give users sometime to migrate
to dynamic reconfig while they can continue use the rolling restarts.
>
> I'd think the ACL control and admin only API should be sufficient for
> security and would prefer to get rid of the flag. But if you must have it,
> we have to prevent both in memory config updates (most important) and
> config file updates if reconfig is disabled. This sounds like a small
> change in quorumpeer, but perhaps I'm forgetting something.
>
Filed https://issues.apache.org/jira/browse/ZOOKEEPER-2819 for this.
>
> Cheers
> Alex
>
>
> On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com> wrote:
>
> > Hi Alex, thanks for clarification!
> >
> > It makes sense to me that users should use reconfig instead of rolling
> > upgrade moving forward. The only concern is backward compatibility now we
> > drop the old rolling upgrade support: since 3.5.x needs to be backward
> > compatible with 3.4.x [1], I think we probably need support rolling
> upgrade
> > for 3.5 branch.
> >
> > As for this flag - I believe it's there and set to false because reconfig
> > is a security sensitive feature and for such features user has to opt in
> > explicitly. Our security team here also has similar recommendations when
> I
> > talked with them about what this feature could do. There are also some
> > discussions around this flag / why it's there in ZOOKEEPER-2014.
> >
> > [1]
> > https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> >
> >
> > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > Hi Michael,
> > >
> > > The described behavior is the intended one - in 3.5 configuration is
> part
> > > of the synced state and is updated
> > > when the server syncs with the leader. The only rolling upgrade I
> tested
> > > was to upgrade the software version
> > > of the servers - this should still work. But I didn't try to support
> > > rolling upgrade for upgrading the configuration,
> > > since this should be done through reconfig.
> > >
> > > I'm still not sure what's the purpose of this flag btw. Why would
> someone
> > > want to do rolling restarts which are prone
> > > to inconsistencies and data loss, when they can use reconfig ?
> > >
> > > Alex
> > >
> > >
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com>
> wrote:
> > >
> > > > reconfigEnabled only disables reconfig command when
> > > reconfigEnabled=false;
> > > > it does not disable the feature by mute all code paths of the
> reconfig
> > > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > > reconfigEnabled,
> > > > 3.5.x ZK will create static config file and dynamic config file in
> any
> > > > cases.
> > > >
> > > > This might create a problem for users who want to do rolling upgrade
> > the
> > > > old way - because now the critical config information is not stored
> in
> > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not work
> > > > because a reconfig needs to go through quorum processors. I think
> this
> > is
> > > > the problem described in the thread.
> > > >
> > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > anything
> > > > mentioned in ZOOKEEPER-107 about this. Currently I think the answer
> is
> > > no,
> > > > which means for 3.5.x the only way to change membership of cluster is
> > > > through reconfig. Could you confirm this conclusion? If that is the
> > case
> > > we
> > > > need patch the reconfigEnabled so it completely disable all code path
> > of
> > > > the reconfig feature to leave the static zoo.cfg intact.
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer <shralex@gmail.com
> >
> > > > wrote:
> > > >
> > > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > > Could you please open a JIRA with the description you provided ?
> > > > >
> > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > exactly
> > > > > to perform the changes you're trying to make, in a simple and
> correct
> > > > way.
> > > > >
> > > > > Thanks,
> > > > > Alex
> > > > >
> > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > gvegator@us.ibm.com>
> > > > > wrote:
> > > > >
> > > > > > I'm still unable to make configuration changes when
> > > > reconfigEnabled=false
> > > > > > by updating zoo.cfg and restarting the servers.
> > > > > >
> > > > > > For example, I want to change the weight of one of my servers. I
> > edit
> > > > > > zoo.cfg on the server I want to change, and specify the group,
> > > > server.x,
> > > > > > and weight.x properties for all servers. I also remove the
> > > > > > dynamicConfigFile property and delete the dynamic config file. I
> > then
> > > > > > restart the server. As soon as the server starts, the dynamic
> > config
> > > > file
> > > > > > re-appears, and it has the last configuration, as if the changes
> I
> > > made
> > > > > in
> > > > > > zoo.cfg were ignored. The dynamic configuration file on the other
> > > > servers
> > > > > > also stays the same.
> > > > > >
> > > > > > What would be the correct way to achieve this (change a server's
> > > > weight,
> > > > > > or role) when reconfigEnabled=false and the CLI reconfig command
> > > cannot
> > > > > be
> > > > > > used?
> > > > > >
> > > > > > Thanks
> > > > > >
> > > > > >
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>
--
Cheers
Michael.
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with
reconfigEnabled=false
Posted by Guillermo Vega-Toro <gv...@us.ibm.com>.
To summarize the issue:
1. A whole ensembe outage is required to modify the ensemble without the
reconfig CLI (stop all servers, modify zoo.cfg, restart servers one by
one).
2. Even with the reconfig CLI, if there is no quorum, it is not possible
to re-configue the ensemble, so one has to fall back to modify the
ensemble through modification of zoo.cfg and restart.
Are these things that can be fixed?
#1 is a change in behavior from 3.4.6. We did not need to stop the whole
ensemble in 3.4.6 to modify the ensemble configuration. We could restart
servers one by one, and eventually a new quorum would be formed.
#2 may be an oversight in the implementation of reconfig. Consider the
case where you have a 5-node ensemble, and 3 nodes go down. There is no
quorum. So, it is not possible to use the reconfig command to for example,
temporarily give a weight of 0 to the failed servers, and a weight of 1 to
the surviving servers, which demotes the ensembe to just 2 nodes until the
outage is resolved. It is necessary to fall back to re-configure the
ensemble through editing zoo.cfg and restarting.
Thanks
"Guillermo Vega-Toro" <gv...@us.ibm.com> wrote on 06/23/2017 09:18:01
AM:
> From: "Guillermo Vega-Toro" <gv...@us.ibm.com>
> To: user@zookeeper.apache.org
> Date: 06/23/2017 09:18 AM
> Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> with reconfigEnabled=false
>
> Thanks all for looking at this.
>
> Here is what I've found to make ensemble config changes work with
> 3.5.3-beta and reconfigEnabled=false:
>
> 1. All servers must be stopped.
> 2. In one server, make the desired changes to zoo.cfg (group, server.x,
> weight.x), and delete the dynamicConfigFile property.
> 3. Start the one server above. You will notice that when the server
> starts, no dynamic file is created, as if the config is "on hold".
> 4. In another server, make the same changes to zoo.cfg, and start the
> server. If there is no quorum, no dynamic file will be created.
> 5. Repeat step 4 for all servers
>
> Once a quorum is reached, the proposed config changes are applied, and a
> dynamic config file will appear on the servers. If you run a 'config'
> command on the zkCLI, the desired configuration will show.
>
> Also, once quorum is reached, further ensemble changes to zoo.cfg
followed
> by restart of a single server are ignored. It is necessary to stop all
> servers and do the steps above to make any changes to the ensemble.
>
> Thanks,
>
> Alexander Shraer <sh...@gmail.com> wrote on 06/23/2017 01:20:47 AM:
>
> > From: Alexander Shraer <sh...@gmail.com>
> > To: user@zookeeper.apache.org
> > Date: 06/23/2017 01:21 AM
> > Subject: Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble
> > with reconfigEnabled=false
> >
> > I'm not sure it's necessary for backward compatibility since rolling
> > restarts for config changes are not really an api the system provides.
> >
> > I'd think the ACL control and admin only API should be sufficient for
> > security and would prefer to get rid of the flag. But if you must have
> it,
> > we have to prevent both in memory config updates (most important) and
> > config file updates if reconfig is disabled. This sounds like a small
> > change in quorumpeer, but perhaps I'm forgetting something.
> >
> > Cheers
> > Alex
> >
> >
> > On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com>
wrote:
> >
> > > Hi Alex, thanks for clarification!
> > >
> > > It makes sense to me that users should use reconfig instead of
rolling
> > > upgrade moving forward. The only concern is backward compatibility
now
> we
> > > drop the old rolling upgrade support: since 3.5.x needs to be
backward
> > > compatible with 3.4.x [1], I think we probably need support rolling
> upgrade
> > > for 3.5 branch.
> > >
> > > As for this flag - I believe it's there and set to false because
> reconfig
> > > is a security sensitive feature and for such features user has to
opt
> in
> > > explicitly. Our security team here also has similar recommendations
> when I
> > > talked with them about what this feature could do. There are also
some
> > > discussions around this flag / why it's there in ZOOKEEPER-2014.
> > >
> > > [1]
> > >
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
> > >
> > >
> > > On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer
<sh...@gmail.com>
> > > wrote:
> > >
> > > > Hi Michael,
> > > >
> > > > The described behavior is the intended one - in 3.5 configuration
is
> part
> > > > of the synced state and is updated
> > > > when the server syncs with the leader. The only rolling upgrade I
> tested
> > > > was to upgrade the software version
> > > > of the servers - this should still work. But I didn't try to
support
> > > > rolling upgrade for upgrading the configuration,
> > > > since this should be done through reconfig.
> > > >
> > > > I'm still not sure what's the purpose of this flag btw. Why would
> someone
> > > > want to do rolling restarts which are prone
> > > > to inconsistencies and data loss, when they can use reconfig ?
> > > >
> > > > Alex
> > > >
> > > >
> > > >
> > > >
> > > > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com>
> wrote:
> > > >
> > > > > reconfigEnabled only disables reconfig command when
> > > > reconfigEnabled=false;
> > > > > it does not disable the feature by mute all code paths of the
> reconfig
> > > > > feature introduced in ZOOKEEPER-107. So regardless of the value
of
> > > > > reconfigEnabled,
> > > > > 3.5.x ZK will create static config file and dynamic config file
in
> any
> > > > > cases.
> > > > >
> > > > > This might create a problem for users who want to do rolling
> upgrade
> > > the
> > > > > old way - because now the critical config information is not
> stored in
> > > > > zoo.cfg anymore and modifying cfg.dynamic file manually will not
> work
> > > > > because a reconfig needs to go through quorum processors. I
think
> this
> > > is
> > > > > the problem described in the thread.
> > > > >
> > > > > Alex, is reconfig compatible with rolling upgrade? I don't find
> > > anything
> > > > > mentioned in ZOOKEEPER-107 about this. Currently I think the
> answer is
> > > > no,
> > > > > which means for 3.5.x the only way to change membership of
cluster
> is
> > > > > through reconfig. Could you confirm this conclusion? If that is
> the
> > > case
> > > > we
> > > > > need patch the reconfigEnabled so it completely disable all code
> path
> > > of
> > > > > the reconfig feature to leave the static zoo.cfg intact.
> > > > >
> > > > >
> > > > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer
> <sh...@gmail.com>
> > > > > wrote:
> > > > >
> > > > > > This sounds like a bug in the implementation of
reconfigEnabled.
> > > > > > Could you please open a JIRA with the description you provided
?
> > > > > >
> > > > > > Out of curiosity, why do you disable reconfig ? It is intended
> > > exactly
> > > > > > to perform the changes you're trying to make, in a simple and
> correct
> > > > > way.
> > > > > >
> > > > > > Thanks,
> > > > > > Alex
> > > > > >
> > > > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > > > gvegator@us.ibm.com>
> > > > > > wrote:
> > > > > >
> > > > > > > I'm still unable to make configuration changes when
> > > > > reconfigEnabled=false
> > > > > > > by updating zoo.cfg and restarting the servers.
> > > > > > >
> > > > > > > For example, I want to change the weight of one of my
servers.
> I
> > > edit
> > > > > > > zoo.cfg on the server I want to change, and specify the
group,
> > > > > server.x,
> > > > > > > and weight.x properties for all servers. I also remove the
> > > > > > > dynamicConfigFile property and delete the dynamic config
file.
> I
> > > then
> > > > > > > restart the server. As soon as the server starts, the
dynamic
> > > config
> > > > > file
> > > > > > > re-appears, and it has the last configuration, as if the
> changes I
> > > > made
> > > > > > in
> > > > > > > zoo.cfg were ignored. The dynamic configuration file on the
> other
> > > > > servers
> > > > > > > also stays the same.
> > > > > > >
> > > > > > > What would be the correct way to achieve this (change a
> server's
> > > > > weight,
> > > > > > > or role) when reconfigEnabled=false and the CLI reconfig
> command
> > > > cannot
> > > > > > be
> > > > > > > used?
> > > > > > >
> > > > > > > Thanks
> > > > > > >
> > > > > > >
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Cheers
> > > > > Michael.
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
>
>
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Alexander Shraer <sh...@gmail.com>.
I'm not sure it's necessary for backward compatibility since rolling
restarts for config changes are not really an api the system provides.
I'd think the ACL control and admin only API should be sufficient for
security and would prefer to get rid of the flag. But if you must have it,
we have to prevent both in memory config updates (most important) and
config file updates if reconfig is disabled. This sounds like a small
change in quorumpeer, but perhaps I'm forgetting something.
Cheers
Alex
On Thu, Jun 22, 2017 at 11:06 PM Michael Han <ha...@cloudera.com> wrote:
> Hi Alex, thanks for clarification!
>
> It makes sense to me that users should use reconfig instead of rolling
> upgrade moving forward. The only concern is backward compatibility now we
> drop the old rolling upgrade support: since 3.5.x needs to be backward
> compatible with 3.4.x [1], I think we probably need support rolling upgrade
> for 3.5 branch.
>
> As for this flag - I believe it's there and set to false because reconfig
> is a security sensitive feature and for such features user has to opt in
> explicitly. Our security team here also has similar recommendations when I
> talked with them about what this feature could do. There are also some
> discussions around this flag / why it's there in ZOOKEEPER-2014.
>
> [1]
> https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
>
>
> On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > Hi Michael,
> >
> > The described behavior is the intended one - in 3.5 configuration is part
> > of the synced state and is updated
> > when the server syncs with the leader. The only rolling upgrade I tested
> > was to upgrade the software version
> > of the servers - this should still work. But I didn't try to support
> > rolling upgrade for upgrading the configuration,
> > since this should be done through reconfig.
> >
> > I'm still not sure what's the purpose of this flag btw. Why would someone
> > want to do rolling restarts which are prone
> > to inconsistencies and data loss, when they can use reconfig ?
> >
> > Alex
> >
> >
> >
> >
> > On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com> wrote:
> >
> > > reconfigEnabled only disables reconfig command when
> > reconfigEnabled=false;
> > > it does not disable the feature by mute all code paths of the reconfig
> > > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > > reconfigEnabled,
> > > 3.5.x ZK will create static config file and dynamic config file in any
> > > cases.
> > >
> > > This might create a problem for users who want to do rolling upgrade
> the
> > > old way - because now the critical config information is not stored in
> > > zoo.cfg anymore and modifying cfg.dynamic file manually will not work
> > > because a reconfig needs to go through quorum processors. I think this
> is
> > > the problem described in the thread.
> > >
> > > Alex, is reconfig compatible with rolling upgrade? I don't find
> anything
> > > mentioned in ZOOKEEPER-107 about this. Currently I think the answer is
> > no,
> > > which means for 3.5.x the only way to change membership of cluster is
> > > through reconfig. Could you confirm this conclusion? If that is the
> case
> > we
> > > need patch the reconfigEnabled so it completely disable all code path
> of
> > > the reconfig feature to leave the static zoo.cfg intact.
> > >
> > >
> > > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer <sh...@gmail.com>
> > > wrote:
> > >
> > > > This sounds like a bug in the implementation of reconfigEnabled.
> > > > Could you please open a JIRA with the description you provided ?
> > > >
> > > > Out of curiosity, why do you disable reconfig ? It is intended
> exactly
> > > > to perform the changes you're trying to make, in a simple and correct
> > > way.
> > > >
> > > > Thanks,
> > > > Alex
> > > >
> > > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > > gvegator@us.ibm.com>
> > > > wrote:
> > > >
> > > > > I'm still unable to make configuration changes when
> > > reconfigEnabled=false
> > > > > by updating zoo.cfg and restarting the servers.
> > > > >
> > > > > For example, I want to change the weight of one of my servers. I
> edit
> > > > > zoo.cfg on the server I want to change, and specify the group,
> > > server.x,
> > > > > and weight.x properties for all servers. I also remove the
> > > > > dynamicConfigFile property and delete the dynamic config file. I
> then
> > > > > restart the server. As soon as the server starts, the dynamic
> config
> > > file
> > > > > re-appears, and it has the last configuration, as if the changes I
> > made
> > > > in
> > > > > zoo.cfg were ignored. The dynamic configuration file on the other
> > > servers
> > > > > also stays the same.
> > > > >
> > > > > What would be the correct way to achieve this (change a server's
> > > weight,
> > > > > or role) when reconfigEnabled=false and the CLI reconfig command
> > cannot
> > > > be
> > > > > used?
> > > > >
> > > > > Thanks
> > > > >
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
> >
>
>
>
> --
> Cheers
> Michael.
>
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
Hi Alex, thanks for clarification!
It makes sense to me that users should use reconfig instead of rolling
upgrade moving forward. The only concern is backward compatibility now we
drop the old rolling upgrade support: since 3.5.x needs to be backward
compatible with 3.4.x [1], I think we probably need support rolling upgrade
for 3.5 branch.
As for this flag - I believe it's there and set to false because reconfig
is a security sensitive feature and for such features user has to opt in
explicitly. Our security team here also has similar recommendations when I
talked with them about what this feature could do. There are also some
discussions around this flag / why it's there in ZOOKEEPER-2014.
[1] https://cwiki.apache.org/confluence/display/ZOOKEEPER/ReleaseManagement
On Thu, Jun 22, 2017 at 10:39 PM, Alexander Shraer <sh...@gmail.com>
wrote:
> Hi Michael,
>
> The described behavior is the intended one - in 3.5 configuration is part
> of the synced state and is updated
> when the server syncs with the leader. The only rolling upgrade I tested
> was to upgrade the software version
> of the servers - this should still work. But I didn't try to support
> rolling upgrade for upgrading the configuration,
> since this should be done through reconfig.
>
> I'm still not sure what's the purpose of this flag btw. Why would someone
> want to do rolling restarts which are prone
> to inconsistencies and data loss, when they can use reconfig ?
>
> Alex
>
>
>
>
> On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com> wrote:
>
> > reconfigEnabled only disables reconfig command when
> reconfigEnabled=false;
> > it does not disable the feature by mute all code paths of the reconfig
> > feature introduced in ZOOKEEPER-107. So regardless of the value of
> > reconfigEnabled,
> > 3.5.x ZK will create static config file and dynamic config file in any
> > cases.
> >
> > This might create a problem for users who want to do rolling upgrade the
> > old way - because now the critical config information is not stored in
> > zoo.cfg anymore and modifying cfg.dynamic file manually will not work
> > because a reconfig needs to go through quorum processors. I think this is
> > the problem described in the thread.
> >
> > Alex, is reconfig compatible with rolling upgrade? I don't find anything
> > mentioned in ZOOKEEPER-107 about this. Currently I think the answer is
> no,
> > which means for 3.5.x the only way to change membership of cluster is
> > through reconfig. Could you confirm this conclusion? If that is the case
> we
> > need patch the reconfigEnabled so it completely disable all code path of
> > the reconfig feature to leave the static zoo.cfg intact.
> >
> >
> > On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > This sounds like a bug in the implementation of reconfigEnabled.
> > > Could you please open a JIRA with the description you provided ?
> > >
> > > Out of curiosity, why do you disable reconfig ? It is intended exactly
> > > to perform the changes you're trying to make, in a simple and correct
> > way.
> > >
> > > Thanks,
> > > Alex
> > >
> > > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> > gvegator@us.ibm.com>
> > > wrote:
> > >
> > > > I'm still unable to make configuration changes when
> > reconfigEnabled=false
> > > > by updating zoo.cfg and restarting the servers.
> > > >
> > > > For example, I want to change the weight of one of my servers. I edit
> > > > zoo.cfg on the server I want to change, and specify the group,
> > server.x,
> > > > and weight.x properties for all servers. I also remove the
> > > > dynamicConfigFile property and delete the dynamic config file. I then
> > > > restart the server. As soon as the server starts, the dynamic config
> > file
> > > > re-appears, and it has the last configuration, as if the changes I
> made
> > > in
> > > > zoo.cfg were ignored. The dynamic configuration file on the other
> > servers
> > > > also stays the same.
> > > >
> > > > What would be the correct way to achieve this (change a server's
> > weight,
> > > > or role) when reconfigEnabled=false and the CLI reconfig command
> cannot
> > > be
> > > > used?
> > > >
> > > > Thanks
> > > >
> > > >
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>
--
Cheers
Michael.
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
On Fri, Jun 23, 2017 at 6:09 AM, Shawn Heisey <ap...@elyograg.org> wrote:
> On 6/22/2017 11:39 PM, Alexander Shraer wrote:
> > The described behavior is the intended one - in 3.5 configuration is
> > part of the synced state and is updated when the server syncs with the
> > leader. The only rolling upgrade I tested was to upgrade the software
> > version of the servers - this should still work. But I didn't try to
> > support rolling upgrade for upgrading the configuration, since this
> > should be done through reconfig.
>
> If the intent is to get rid of the old way of changing the configuration
> (update zoo.cfg and perform rolling restarts) and only support dynamic
> reconfiguration, then why is there a reconfigEnabled setting at all, and
> why does it default to false?
I don't think the intention is to completely deprecated the rolling
restarts in favor of reconfig - at least for 3.5.x releases. There was no
record on JIRA / apache email about the decision on deprecating rolling
restart as far as I can tell. I think the issue is that we are not very
well aware of this problem until OP brought this up.
> Based on everything that has been said here, it sounds like when
> reconfigEnabled is left alone or explicitly set it to false, the ability
> to change the ensemble configuration is entirely lost, because the
> server will pull the dynamic config from the ensemble on startup and any
> changes made to zoo.cfg are ignored.
>
This is correct.
>
> Thanks,
> Shawn
>
>
--
Cheers
Michael.
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with
reconfigEnabled=false
Posted by Shawn Heisey <ap...@elyograg.org>.
On 6/22/2017 11:39 PM, Alexander Shraer wrote:
> The described behavior is the intended one - in 3.5 configuration is
> part of the synced state and is updated when the server syncs with the
> leader. The only rolling upgrade I tested was to upgrade the software
> version of the servers - this should still work. But I didn't try to
> support rolling upgrade for upgrading the configuration, since this
> should be done through reconfig.
If the intent is to get rid of the old way of changing the configuration
(update zoo.cfg and perform rolling restarts) and only support dynamic
reconfiguration, then why is there a reconfigEnabled setting at all, and
why does it default to false?
Based on everything that has been said here, it sounds like when
reconfigEnabled is left alone or explicitly set it to false, the ability
to change the ensemble configuration is entirely lost, because the
server will pull the dynamic config from the ensemble on startup and any
changes made to zoo.cfg are ignored.
Thanks,
Shawn
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Alexander Shraer <sh...@gmail.com>.
Hi Michael,
The described behavior is the intended one - in 3.5 configuration is part
of the synced state and is updated
when the server syncs with the leader. The only rolling upgrade I tested
was to upgrade the software version
of the servers - this should still work. But I didn't try to support
rolling upgrade for upgrading the configuration,
since this should be done through reconfig.
I'm still not sure what's the purpose of this flag btw. Why would someone
want to do rolling restarts which are prone
to inconsistencies and data loss, when they can use reconfig ?
Alex
On Thu, Jun 22, 2017 at 10:18 PM, Michael Han <ha...@cloudera.com> wrote:
> reconfigEnabled only disables reconfig command when reconfigEnabled=false;
> it does not disable the feature by mute all code paths of the reconfig
> feature introduced in ZOOKEEPER-107. So regardless of the value of
> reconfigEnabled,
> 3.5.x ZK will create static config file and dynamic config file in any
> cases.
>
> This might create a problem for users who want to do rolling upgrade the
> old way - because now the critical config information is not stored in
> zoo.cfg anymore and modifying cfg.dynamic file manually will not work
> because a reconfig needs to go through quorum processors. I think this is
> the problem described in the thread.
>
> Alex, is reconfig compatible with rolling upgrade? I don't find anything
> mentioned in ZOOKEEPER-107 about this. Currently I think the answer is no,
> which means for 3.5.x the only way to change membership of cluster is
> through reconfig. Could you confirm this conclusion? If that is the case we
> need patch the reconfigEnabled so it completely disable all code path of
> the reconfig feature to leave the static zoo.cfg intact.
>
>
> On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > This sounds like a bug in the implementation of reconfigEnabled.
> > Could you please open a JIRA with the description you provided ?
> >
> > Out of curiosity, why do you disable reconfig ? It is intended exactly
> > to perform the changes you're trying to make, in a simple and correct
> way.
> >
> > Thanks,
> > Alex
> >
> > On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <
> gvegator@us.ibm.com>
> > wrote:
> >
> > > I'm still unable to make configuration changes when
> reconfigEnabled=false
> > > by updating zoo.cfg and restarting the servers.
> > >
> > > For example, I want to change the weight of one of my servers. I edit
> > > zoo.cfg on the server I want to change, and specify the group,
> server.x,
> > > and weight.x properties for all servers. I also remove the
> > > dynamicConfigFile property and delete the dynamic config file. I then
> > > restart the server. As soon as the server starts, the dynamic config
> file
> > > re-appears, and it has the last configuration, as if the changes I made
> > in
> > > zoo.cfg were ignored. The dynamic configuration file on the other
> servers
> > > also stays the same.
> > >
> > > What would be the correct way to achieve this (change a server's
> weight,
> > > or role) when reconfigEnabled=false and the CLI reconfig command cannot
> > be
> > > used?
> > >
> > > Thanks
> > >
> > >
> >
>
>
>
> --
> Cheers
> Michael.
>
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
reconfigEnabled only disables reconfig command when reconfigEnabled=false;
it does not disable the feature by mute all code paths of the reconfig
feature introduced in ZOOKEEPER-107. So regardless of the value of
reconfigEnabled,
3.5.x ZK will create static config file and dynamic config file in any
cases.
This might create a problem for users who want to do rolling upgrade the
old way - because now the critical config information is not stored in
zoo.cfg anymore and modifying cfg.dynamic file manually will not work
because a reconfig needs to go through quorum processors. I think this is
the problem described in the thread.
Alex, is reconfig compatible with rolling upgrade? I don't find anything
mentioned in ZOOKEEPER-107 about this. Currently I think the answer is no,
which means for 3.5.x the only way to change membership of cluster is
through reconfig. Could you confirm this conclusion? If that is the case we
need patch the reconfigEnabled so it completely disable all code path of
the reconfig feature to leave the static zoo.cfg intact.
On Thu, Jun 22, 2017 at 9:35 PM, Alexander Shraer <sh...@gmail.com> wrote:
> This sounds like a bug in the implementation of reconfigEnabled.
> Could you please open a JIRA with the description you provided ?
>
> Out of curiosity, why do you disable reconfig ? It is intended exactly
> to perform the changes you're trying to make, in a simple and correct way.
>
> Thanks,
> Alex
>
> On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <gv...@us.ibm.com>
> wrote:
>
> > I'm still unable to make configuration changes when reconfigEnabled=false
> > by updating zoo.cfg and restarting the servers.
> >
> > For example, I want to change the weight of one of my servers. I edit
> > zoo.cfg on the server I want to change, and specify the group, server.x,
> > and weight.x properties for all servers. I also remove the
> > dynamicConfigFile property and delete the dynamic config file. I then
> > restart the server. As soon as the server starts, the dynamic config file
> > re-appears, and it has the last configuration, as if the changes I made
> in
> > zoo.cfg were ignored. The dynamic configuration file on the other servers
> > also stays the same.
> >
> > What would be the correct way to achieve this (change a server's weight,
> > or role) when reconfigEnabled=false and the CLI reconfig command cannot
> be
> > used?
> >
> > Thanks
> >
> >
>
--
Cheers
Michael.
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Alexander Shraer <sh...@gmail.com>.
This sounds like a bug in the implementation of reconfigEnabled.
Could you please open a JIRA with the description you provided ?
Out of curiosity, why do you disable reconfig ? It is intended exactly
to perform the changes you're trying to make, in a simple and correct way.
Thanks,
Alex
On Thu, Jun 22, 2017 at 3:17 PM, Guillermo Vega-Toro <gv...@us.ibm.com>
wrote:
> I'm still unable to make configuration changes when reconfigEnabled=false
> by updating zoo.cfg and restarting the servers.
>
> For example, I want to change the weight of one of my servers. I edit
> zoo.cfg on the server I want to change, and specify the group, server.x,
> and weight.x properties for all servers. I also remove the
> dynamicConfigFile property and delete the dynamic config file. I then
> restart the server. As soon as the server starts, the dynamic config file
> re-appears, and it has the last configuration, as if the changes I made in
> zoo.cfg were ignored. The dynamic configuration file on the other servers
> also stays the same.
>
> What would be the correct way to achieve this (change a server's weight,
> or role) when reconfigEnabled=false and the CLI reconfig command cannot be
> used?
>
> Thanks
>
>
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with
reconfigEnabled=false
Posted by Guillermo Vega-Toro <gv...@us.ibm.com>.
I'm still unable to make configuration changes when reconfigEnabled=false
by updating zoo.cfg and restarting the servers.
For example, I want to change the weight of one of my servers. I edit
zoo.cfg on the server I want to change, and specify the group, server.x,
and weight.x properties for all servers. I also remove the
dynamicConfigFile property and delete the dynamic config file. I then
restart the server. As soon as the server starts, the dynamic config file
re-appears, and it has the last configuration, as if the changes I made in
zoo.cfg were ignored. The dynamic configuration file on the other servers
also stays the same.
What would be the correct way to achieve this (change a server's weight,
or role) when reconfigEnabled=false and the CLI reconfig command cannot be
used?
Thanks
Re: How to add nodes to a Zookeeper 3.5.3-beta ensemble with reconfigEnabled=false
Posted by Michael Han <ha...@cloudera.com>.
You can still do rolling restarts for 3.5.x including 3.5.3-beta. Rolling
restart requires edits the zoo.cfg - the static configuration files,
instead of zoo.cfg.dynamic.x, which is the dynamic reconfiguration file
that stores reconfig parameters. This dynamic config file is managed by ZK
and is not supposed to be edited by users directly. See
http://zookeeper.apache.org/doc/r3.5.3-beta/zookeeperReconfig.html#sc_
reconfig_file for more details.
Another option is to use dynamic reconfig through command line, the doc
linked above should provide enough information to get started.
On Wed, Jun 21, 2017 at 11:33 AM, Guillermo Vega-Toro <gv...@us.ibm.com>
wrote:
> Hi,
>
> I have a 3-node Zookeeper 3.5.3-beta ensemble. This was initially created
> by un-taring the Zookeeper tarball into 3 machines, editing the zoo.cfg
> file with the 3 server entries, and starting the servers. The ensemble
> started, and a zoo.cfg.dynamic.100000000 file was created with the
> server.x lines in all servers.
>
> Now, I want to add 2 more nodes to grow to a 5-node ensemble. In Zookeeper
> 3.4.6, I would install the two new nodes, update the zoo.cfg files on all
> servers to add the new nodes, and restart the servers one by one. But in
> version 3.5.3-beta, this is not possible - any changes made to
> zoo.cfg.dynamic.100000000 are ignored, even if reconfiguration is
> disabled. If the dynamic config file is modified to add new servers and
> the server is restarted, the dynamic config file gets overwritten with the
> old 3-node configuration when the server starts.
>
> So, it looks like in Zookeeper 3.5.x, the way to add nodes is to use the
> new reconfig command on the CLI. But if the reconfiguration feature is
> disabled for security reasons (as it is in 3.5.3-beta by default), then
> adding nodes "the old way" by editing the config file and restarting the
> servers is not possible either. Does this mean that in Zookeeper
> 3.5.3-beta, in order to grow the ensemble, it is necessary to enable
> reconfiguration (by setting reconfigEnabled=true in zoo.cfg and restarting
> all servers), then running the CLI to add the new nodes? Then after the
> new nodes are added, edit zoo.cfg to disable reconfiguration on all
> servers, and restart all servers again?
>
> Thanks
>
>
>
--
Cheers
Michael.