You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Nomar Morado <no...@gmail.com> on 2016/06/14 15:53:35 UTC
observer changing to participant when there is no quorum
Hi
I was trying to promote an observer into participant when ZK loses quorum -
but it seems that it does not allow to.
Would you know how this can be accomplished without having to recycle ZK?
I am using 3.5.0-alpha
Thanks.
Re: observer changing to participant when there is no quorum
Posted by Alexander Shraer <sh...@gmail.com>.
ZooKeeper only works when a majority of the participants are up.
Since 2 out of 3 participants in your ensemble are down, ZooKeeper won't
allow you to issue any commands, including a reconfiguration.
You should have enough participants such that a situation where a majority
is simultaneously down doesn't happen.
On Wed, Jun 15, 2016 at 4:33 PM, Nomar Morado <j3...@icloud.com>
wrote:
> Corrected typos
>
> Printing e-mails wastes valuable natural resources. Please don't print
> this message unless it is absolutely necessary. Thank you for thinking
> green!
>
> Sent from my iPhone
>
> > On Jun 15, 2016, at 9:31 AM, Nomar Morado <j3...@icloud.com>
> wrote:
> >
> > This is what I have done so far:
> >
> > A,B,C are participants
> > C,D are observers
> >
> > B,C are offline (crashed)
> >
> > I am trying to:
> >
> > 1. Remove C, D
> > 2. Add C,D back as participants
> >
> > Will this work?
> >
> > At least in my testing (might be doing wrong) I am getting this error on
> the first step and hence can't get forward:
> >
> > Client could not connect to reestablished quorum: giving up after 30+
> seconds
> >
> >
> > I am passing the original server configure string to zk's reconfig
> method.
> >
> >
> >
> > Thanks
> >
> >
> > Printing e-mails wastes valuable natural resources. Please don't print
> this message unless it is absolutely necessary. Thank you for thinking
> green!
> >
> > Sent from my iPhone
> >
> >> On Jun 14, 2016, at 10:55 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
> >>
> >> Right, a quorum of participants from the old config is required to
> process
> >> any command, including reconfig,
> >> and a quorum of participants from the new config is required for the
> >> reconfig to even start. If there's no such connected
> >> quorum an error NewConfigNoQuorum will be thrown.
> >>
> >> But there is one slightly confusing case where the error is thrown,
> which
> >> is explained in the doc: when you are
> >> converting an observer to a participant and there is no quorum in the
> new
> >> config without counting that "future" participant.
> >> So the server is connected, but since its not a participant we get the
> >> error above. In that case, one first needs to
> >> convert the observer to remove the observer and then add it back. The
> >> detailed explanation is in the doc, look for
> >> "Changing an observer into a follower".
> >>
> >> On Wed, Jun 15, 2016 at 1:17 AM, Camille Fournier <ca...@apache.org>
> >> wrote:
> >>
> >>> I'm finding the documentation quite confusing. I was under the
> impression
> >>> that quorum of some sort was needed to do a reconfig. Can you reconfig
> when
> >>> there is no quorum?
> >>>
> >>> *Progress guarantees:* Up to the invocation of the reconfig operation,
> a
> >>> quorum of the old configuration is required to be available and
> connected
> >>> for ZooKeeper to be able to make progress. Once reconfig is invoked, a
> >>> quorum of both the old and of the new configurations must be available.
> >>>
> >>> *Adding servers:* Before a reconfiguration is invoked, the
> administrator
> >>> must make sure that a quorum (majority) of participants from the new
> >>> configuration are already connected and synced with the current leader.
> >>>
> >>>
> >>>
> >>> On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com>
> >>> wrote:
> >>>
> >>>> This is needed only in case the target config doesn't have a quorum
> which
> >>>> are already followers in the old config
> >>>> and are up. We need agreement from a quorum of the target config, but
> >>>> observers aren't participating
> >>>> in the voting protocol.
> >>>>
> >>>>> On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com>
> wrote:
> >>>>>
> >>>>> This might help:
> >>>>> https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html
> section
> >>>>> '*Changing
> >>>>> an observer into a follower:'*
> >>>>> "first invoke a reconfig to remove D from the configuration and then
> >>>> invoke
> >>>>> a second command to add it back as a participant (follower)."
> >>>>>
> >>>>>
> >>>>> On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <
> nomar.morado@gmail.com>
> >>>>> wrote:
> >>>>>
> >>>>>> Hi
> >>>>>>
> >>>>>> I was trying to promote an observer into participant when ZK loses
> >>>>> quorum -
> >>>>>> but it seems that it does not allow to.
> >>>>>>
> >>>>>> Would you know how this can be accomplished without having to
> recycle
> >>>> ZK?
> >>>>>>
> >>>>>> I am using 3.5.0-alpha
> >>>>>>
> >>>>>>
> >>>>>> Thanks.
> >>>>>
> >>>>>
> >>>>>
> >>>>> --
> >>>>> Cheers
> >>>>> Michael.
> >>>
>
Re: observer changing to participant when there is no quorum
Posted by Nomar Morado <j3...@icloud.com>.
Corrected typos
Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green!
Sent from my iPhone
> On Jun 15, 2016, at 9:31 AM, Nomar Morado <j3...@icloud.com> wrote:
>
> This is what I have done so far:
>
> A,B,C are participants
> C,D are observers
>
> B,C are offline (crashed)
>
> I am trying to:
>
> 1. Remove C, D
> 2. Add C,D back as participants
>
> Will this work?
>
> At least in my testing (might be doing wrong) I am getting this error on the first step and hence can't get forward:
>
> Client could not connect to reestablished quorum: giving up after 30+ seconds
>
>
> I am passing the original server configure string to zk's reconfig method.
>
>
>
> Thanks
>
>
> Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green!
>
> Sent from my iPhone
>
>> On Jun 14, 2016, at 10:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>>
>> Right, a quorum of participants from the old config is required to process
>> any command, including reconfig,
>> and a quorum of participants from the new config is required for the
>> reconfig to even start. If there's no such connected
>> quorum an error NewConfigNoQuorum will be thrown.
>>
>> But there is one slightly confusing case where the error is thrown, which
>> is explained in the doc: when you are
>> converting an observer to a participant and there is no quorum in the new
>> config without counting that "future" participant.
>> So the server is connected, but since its not a participant we get the
>> error above. In that case, one first needs to
>> convert the observer to remove the observer and then add it back. The
>> detailed explanation is in the doc, look for
>> "Changing an observer into a follower".
>>
>> On Wed, Jun 15, 2016 at 1:17 AM, Camille Fournier <ca...@apache.org>
>> wrote:
>>
>>> I'm finding the documentation quite confusing. I was under the impression
>>> that quorum of some sort was needed to do a reconfig. Can you reconfig when
>>> there is no quorum?
>>>
>>> *Progress guarantees:* Up to the invocation of the reconfig operation, a
>>> quorum of the old configuration is required to be available and connected
>>> for ZooKeeper to be able to make progress. Once reconfig is invoked, a
>>> quorum of both the old and of the new configurations must be available.
>>>
>>> *Adding servers:* Before a reconfiguration is invoked, the administrator
>>> must make sure that a quorum (majority) of participants from the new
>>> configuration are already connected and synced with the current leader.
>>>
>>>
>>>
>>> On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com>
>>> wrote:
>>>
>>>> This is needed only in case the target config doesn't have a quorum which
>>>> are already followers in the old config
>>>> and are up. We need agreement from a quorum of the target config, but
>>>> observers aren't participating
>>>> in the voting protocol.
>>>>
>>>>> On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com> wrote:
>>>>>
>>>>> This might help:
>>>>> https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
>>>>> '*Changing
>>>>> an observer into a follower:'*
>>>>> "first invoke a reconfig to remove D from the configuration and then
>>>> invoke
>>>>> a second command to add it back as a participant (follower)."
>>>>>
>>>>>
>>>>> On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi
>>>>>>
>>>>>> I was trying to promote an observer into participant when ZK loses
>>>>> quorum -
>>>>>> but it seems that it does not allow to.
>>>>>>
>>>>>> Would you know how this can be accomplished without having to recycle
>>>> ZK?
>>>>>>
>>>>>> I am using 3.5.0-alpha
>>>>>>
>>>>>>
>>>>>> Thanks.
>>>>>
>>>>>
>>>>>
>>>>> --
>>>>> Cheers
>>>>> Michael.
>>>
Re: observer changing to participant when there is no quorum
Posted by Nomar Morado <j3...@icloud.com>.
This is what I have done so far:
A,B,C are participants
C,D are observers
B,C are offline (crashed)
I am trying to:
1. Remove C, D
2. Add C,D back as participants
Will this work?
At least in my testing (might be doing wrong) I am getting this error on the first step and hence can't get forward:
Client could not connect to rests blushed quorum: giving up after 30+ seconds
I am passing the original server configure string to zk's reconfig method.
Thanks
Printing e-mails wastes valuable natural resources. Please don't print this message unless it is absolutely necessary. Thank you for thinking green!
Sent from my iPhone
> On Jun 14, 2016, at 10:55 PM, Alexander Shraer <sh...@gmail.com> wrote:
>
> Right, a quorum of participants from the old config is required to process
> any command, including reconfig,
> and a quorum of participants from the new config is required for the
> reconfig to even start. If there's no such connected
> quorum an error NewConfigNoQuorum will be thrown.
>
> But there is one slightly confusing case where the error is thrown, which
> is explained in the doc: when you are
> converting an observer to a participant and there is no quorum in the new
> config without counting that "future" participant.
> So the server is connected, but since its not a participant we get the
> error above. In that case, one first needs to
> convert the observer to remove the observer and then add it back. The
> detailed explanation is in the doc, look for
> "Changing an observer into a follower".
>
> On Wed, Jun 15, 2016 at 1:17 AM, Camille Fournier <ca...@apache.org>
> wrote:
>
>> I'm finding the documentation quite confusing. I was under the impression
>> that quorum of some sort was needed to do a reconfig. Can you reconfig when
>> there is no quorum?
>>
>> *Progress guarantees:* Up to the invocation of the reconfig operation, a
>> quorum of the old configuration is required to be available and connected
>> for ZooKeeper to be able to make progress. Once reconfig is invoked, a
>> quorum of both the old and of the new configurations must be available.
>>
>> *Adding servers:* Before a reconfiguration is invoked, the administrator
>> must make sure that a quorum (majority) of participants from the new
>> configuration are already connected and synced with the current leader.
>>
>>
>>
>> On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com>
>> wrote:
>>
>>> This is needed only in case the target config doesn't have a quorum which
>>> are already followers in the old config
>>> and are up. We need agreement from a quorum of the target config, but
>>> observers aren't participating
>>> in the voting protocol.
>>>
>>>> On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com> wrote:
>>>>
>>>> This might help:
>>>> https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
>>>> '*Changing
>>>> an observer into a follower:'*
>>>> "first invoke a reconfig to remove D from the configuration and then
>>> invoke
>>>> a second command to add it back as a participant (follower)."
>>>>
>>>>
>>>> On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
>>>> wrote:
>>>>
>>>>> Hi
>>>>>
>>>>> I was trying to promote an observer into participant when ZK loses
>>>> quorum -
>>>>> but it seems that it does not allow to.
>>>>>
>>>>> Would you know how this can be accomplished without having to recycle
>>> ZK?
>>>>>
>>>>> I am using 3.5.0-alpha
>>>>>
>>>>>
>>>>> Thanks.
>>>>
>>>>
>>>>
>>>> --
>>>> Cheers
>>>> Michael.
>>
Re: observer changing to participant when there is no quorum
Posted by Camille Fournier <ca...@apache.org>.
Thanks Alex. So if the original questioner is trying to reconfig without
quorum that's not possible, and with quorum you may need to reconfig to
remove the observer entirely then add it back as a voting member.
On Jun 14, 2016 10:55 PM, "Alexander Shraer" <sh...@gmail.com> wrote:
> Right, a quorum of participants from the old config is required to process
> any command, including reconfig,
> and a quorum of participants from the new config is required for the
> reconfig to even start. If there's no such connected
> quorum an error NewConfigNoQuorum will be thrown.
>
> But there is one slightly confusing case where the error is thrown, which
> is explained in the doc: when you are
> converting an observer to a participant and there is no quorum in the new
> config without counting that "future" participant.
> So the server is connected, but since its not a participant we get the
> error above. In that case, one first needs to
> convert the observer to remove the observer and then add it back. The
> detailed explanation is in the doc, look for
> "Changing an observer into a follower".
>
> On Wed, Jun 15, 2016 at 1:17 AM, Camille Fournier <ca...@apache.org>
> wrote:
>
> > I'm finding the documentation quite confusing. I was under the impression
> > that quorum of some sort was needed to do a reconfig. Can you reconfig
> when
> > there is no quorum?
> >
> > *Progress guarantees:* Up to the invocation of the reconfig operation, a
> > quorum of the old configuration is required to be available and connected
> > for ZooKeeper to be able to make progress. Once reconfig is invoked, a
> > quorum of both the old and of the new configurations must be available.
> >
> > *Adding servers:* Before a reconfiguration is invoked, the administrator
> > must make sure that a quorum (majority) of participants from the new
> > configuration are already connected and synced with the current leader.
> >
> >
> >
> > On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com>
> > wrote:
> >
> > > This is needed only in case the target config doesn't have a quorum
> which
> > > are already followers in the old config
> > > and are up. We need agreement from a quorum of the target config, but
> > > observers aren't participating
> > > in the voting protocol.
> > >
> > > On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com>
> wrote:
> > >
> > > > This might help:
> > > > https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html
> section
> > > > '*Changing
> > > > an observer into a follower:'*
> > > > "first invoke a reconfig to remove D from the configuration and then
> > > invoke
> > > > a second command to add it back as a participant (follower)."
> > > >
> > > >
> > > > On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <
> nomar.morado@gmail.com>
> > > > wrote:
> > > >
> > > > > Hi
> > > > >
> > > > > I was trying to promote an observer into participant when ZK loses
> > > > quorum -
> > > > > but it seems that it does not allow to.
> > > > >
> > > > > Would you know how this can be accomplished without having to
> recycle
> > > ZK?
> > > > >
> > > > > I am using 3.5.0-alpha
> > > > >
> > > > >
> > > > > Thanks.
> > > > >
> > > >
> > > >
> > > >
> > > > --
> > > > Cheers
> > > > Michael.
> > > >
> > >
> >
>
Re: observer changing to participant when there is no quorum
Posted by Alexander Shraer <sh...@gmail.com>.
Right, a quorum of participants from the old config is required to process
any command, including reconfig,
and a quorum of participants from the new config is required for the
reconfig to even start. If there's no such connected
quorum an error NewConfigNoQuorum will be thrown.
But there is one slightly confusing case where the error is thrown, which
is explained in the doc: when you are
converting an observer to a participant and there is no quorum in the new
config without counting that "future" participant.
So the server is connected, but since its not a participant we get the
error above. In that case, one first needs to
convert the observer to remove the observer and then add it back. The
detailed explanation is in the doc, look for
"Changing an observer into a follower".
On Wed, Jun 15, 2016 at 1:17 AM, Camille Fournier <ca...@apache.org>
wrote:
> I'm finding the documentation quite confusing. I was under the impression
> that quorum of some sort was needed to do a reconfig. Can you reconfig when
> there is no quorum?
>
> *Progress guarantees:* Up to the invocation of the reconfig operation, a
> quorum of the old configuration is required to be available and connected
> for ZooKeeper to be able to make progress. Once reconfig is invoked, a
> quorum of both the old and of the new configurations must be available.
>
> *Adding servers:* Before a reconfiguration is invoked, the administrator
> must make sure that a quorum (majority) of participants from the new
> configuration are already connected and synced with the current leader.
>
>
>
> On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com>
> wrote:
>
> > This is needed only in case the target config doesn't have a quorum which
> > are already followers in the old config
> > and are up. We need agreement from a quorum of the target config, but
> > observers aren't participating
> > in the voting protocol.
> >
> > On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com> wrote:
> >
> > > This might help:
> > > https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
> > > '*Changing
> > > an observer into a follower:'*
> > > "first invoke a reconfig to remove D from the configuration and then
> > invoke
> > > a second command to add it back as a participant (follower)."
> > >
> > >
> > > On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
> > > wrote:
> > >
> > > > Hi
> > > >
> > > > I was trying to promote an observer into participant when ZK loses
> > > quorum -
> > > > but it seems that it does not allow to.
> > > >
> > > > Would you know how this can be accomplished without having to recycle
> > ZK?
> > > >
> > > > I am using 3.5.0-alpha
> > > >
> > > >
> > > > Thanks.
> > > >
> > >
> > >
> > >
> > > --
> > > Cheers
> > > Michael.
> > >
> >
>
Re: observer changing to participant when there is no quorum
Posted by Camille Fournier <ca...@apache.org>.
I'm finding the documentation quite confusing. I was under the impression
that quorum of some sort was needed to do a reconfig. Can you reconfig when
there is no quorum?
*Progress guarantees:* Up to the invocation of the reconfig operation, a
quorum of the old configuration is required to be available and connected
for ZooKeeper to be able to make progress. Once reconfig is invoked, a
quorum of both the old and of the new configurations must be available.
*Adding servers:* Before a reconfiguration is invoked, the administrator
must make sure that a quorum (majority) of participants from the new
configuration are already connected and synced with the current leader.
On Tue, Jun 14, 2016 at 5:35 PM, Alexander Shraer <sh...@gmail.com> wrote:
> This is needed only in case the target config doesn't have a quorum which
> are already followers in the old config
> and are up. We need agreement from a quorum of the target config, but
> observers aren't participating
> in the voting protocol.
>
> On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com> wrote:
>
> > This might help:
> > https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
> > '*Changing
> > an observer into a follower:'*
> > "first invoke a reconfig to remove D from the configuration and then
> invoke
> > a second command to add it back as a participant (follower)."
> >
> >
> > On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
> > wrote:
> >
> > > Hi
> > >
> > > I was trying to promote an observer into participant when ZK loses
> > quorum -
> > > but it seems that it does not allow to.
> > >
> > > Would you know how this can be accomplished without having to recycle
> ZK?
> > >
> > > I am using 3.5.0-alpha
> > >
> > >
> > > Thanks.
> > >
> >
> >
> >
> > --
> > Cheers
> > Michael.
> >
>
Re: observer changing to participant when there is no quorum
Posted by Alexander Shraer <sh...@gmail.com>.
This is needed only in case the target config doesn't have a quorum which
are already followers in the old config
and are up. We need agreement from a quorum of the target config, but
observers aren't participating
in the voting protocol.
On Tue, Jun 14, 2016 at 7:35 PM, Michael Han <ha...@cloudera.com> wrote:
> This might help:
> https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
> '*Changing
> an observer into a follower:'*
> "first invoke a reconfig to remove D from the configuration and then invoke
> a second command to add it back as a participant (follower)."
>
>
> On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
> wrote:
>
> > Hi
> >
> > I was trying to promote an observer into participant when ZK loses
> quorum -
> > but it seems that it does not allow to.
> >
> > Would you know how this can be accomplished without having to recycle ZK?
> >
> > I am using 3.5.0-alpha
> >
> >
> > Thanks.
> >
>
>
>
> --
> Cheers
> Michael.
>
Re: observer changing to participant when there is no quorum
Posted by Michael Han <ha...@cloudera.com>.
This might help:
https://zookeeper.apache.org/doc/trunk/zookeeperReconfig.html section
'*Changing
an observer into a follower:'*
"first invoke a reconfig to remove D from the configuration and then invoke
a second command to add it back as a participant (follower)."
On Tue, Jun 14, 2016 at 8:53 AM, Nomar Morado <no...@gmail.com>
wrote:
> Hi
>
> I was trying to promote an observer into participant when ZK loses quorum -
> but it seems that it does not allow to.
>
> Would you know how this can be accomplished without having to recycle ZK?
>
> I am using 3.5.0-alpha
>
>
> Thanks.
>
--
Cheers
Michael.