You are viewing a plain text version of this content. The canonical link for it is here.
Posted to user@zookeeper.apache.org by Dillian Murphey <cr...@gmail.com> on 2015/01/29 00:40:40 UTC
Basic 3 ZK, bring 1 down question.
If I have 3 ZK instances running on separate servers and I terminate one of
the servers, what should I expect to happen?
Will the remaining 2 zookeepers continue happily and stay online without
any interruption?
I'm using exhibitor/zookeeper and I'm attempting this scenerio and what I'm
seeing is my remaining 2 zookeepers go offline. I thought the whole point
of having a majority was to allow a failure?
Thank you. I'm new so be kind.
Re: Basic 3 ZK, bring 1 down question.
Posted by Cameron McKenzie <mc...@gmail.com>.
Can't help you out on the Exhibitor front as I haven't used it. I'd have a
look at the configuration files it's generating though as it appears that
some of the nodes are configured as observers rather than fully fledged
node members.
On Thu, Jan 29, 2015 at 11:08 AM, Dillian Murphey <cr...@gmail.com>
wrote:
> Thanks Cameron, ok so I just ran the exercise again. I killed a server.
> The other two ZKs go offline and the zookeeper.out log is as so:
>
> So why does this feel as if I need to manually adjust the config file?
>
> Thanks. P.S. I am using netflix opensource exhibitor. It may be a factor, I
> don't know. The creator seems to brush me off a lot.
>
> 2015-01-29 00:05:51,271 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
> config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
> tail: zookeeper.out: file truncated
> 2015-01-29 00:06:11,561 [myid:] - INFO [main:QuorumPeerConfig@103] -
> Reading configuration from: /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> 2015-01-29 00:06:11,566 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
> config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
>
> On Wed, Jan 28, 2015 at 3:45 PM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
> > Assuming that your 2 remaining servers can communicate with each other
> the
> > cluster should should remain online as it still has a quorum (2 / 3 nodes
> > are online). There may be a brief period where the cluster won't accept
> > requires while a new leadership election occurs between the 2 remaining
> > nodes.
> > cheers
> > Cam
> >
> > On Thu, Jan 29, 2015 at 10:40 AM, Dillian Murphey <
> crackshotmule@gmail.com
> > >
> > wrote:
> >
> > > If I have 3 ZK instances running on separate servers and I terminate
> one
> > of
> > > the servers, what should I expect to happen?
> > >
> > > Will the remaining 2 zookeepers continue happily and stay online
> without
> > > any interruption?
> > >
> > > I'm using exhibitor/zookeeper and I'm attempting this scenerio and what
> > I'm
> > > seeing is my remaining 2 zookeepers go offline. I thought the whole
> > point
> > > of having a majority was to allow a failure?
> > >
> > >
> > > Thank you. I'm new so be kind.
> > >
> >
>
Re: Basic 3 ZK, bring 1 down question.
Posted by Dillian Murphey <cr...@gmail.com>.
Oh, in fact, what happened was, I had 3 ZK nodes, and one was an observer.
When I brought down the one server, I didn't have a quorum (gotta love that
word). So I had to manually change my zoo.cfg and remove the :observer
flag. Then my ZKs came back up.
So what am I doing wrong? I want to make sure a server can go down my ZKs
remain up. :)
thanks again!
On Wed, Jan 28, 2015 at 4:08 PM, Dillian Murphey <cr...@gmail.com>
wrote:
> Thanks Cameron, ok so I just ran the exercise again. I killed a server.
> The other two ZKs go offline and the zookeeper.out log is as so:
>
> So why does this feel as if I need to manually adjust the config file?
>
> Thanks. P.S. I am using netflix opensource exhibitor. It may be a factor,
> I don't know. The creator seems to brush me off a lot.
>
> 2015-01-29 00:05:51,271 [myid:] - ERROR [main:QuorumPeerMain@85] -
> Invalid config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
> tail: zookeeper.out: file truncated
> 2015-01-29 00:06:11,561 [myid:] - INFO [main:QuorumPeerConfig@103] -
> Reading configuration from: /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> 2015-01-29 00:06:11,566 [myid:] - ERROR [main:QuorumPeerMain@85] -
> Invalid config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
>
> On Wed, Jan 28, 2015 at 3:45 PM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
>> Assuming that your 2 remaining servers can communicate with each other the
>> cluster should should remain online as it still has a quorum (2 / 3 nodes
>> are online). There may be a brief period where the cluster won't accept
>> requires while a new leadership election occurs between the 2 remaining
>> nodes.
>> cheers
>> Cam
>>
>> On Thu, Jan 29, 2015 at 10:40 AM, Dillian Murphey <
>> crackshotmule@gmail.com>
>> wrote:
>>
>> > If I have 3 ZK instances running on separate servers and I terminate
>> one of
>> > the servers, what should I expect to happen?
>> >
>> > Will the remaining 2 zookeepers continue happily and stay online without
>> > any interruption?
>> >
>> > I'm using exhibitor/zookeeper and I'm attempting this scenerio and what
>> I'm
>> > seeing is my remaining 2 zookeepers go offline. I thought the whole
>> point
>> > of having a majority was to allow a failure?
>> >
>> >
>> > Thank you. I'm new so be kind.
>> >
>>
>
>
Re: Basic 3 ZK, bring 1 down question.
Posted by John Sirois <jo...@gmail.com>.
You probably need to include your config at this point to debug further.
On Wed, Jan 28, 2015 at 5:08 PM, Dillian Murphey <cr...@gmail.com>
wrote:
> Thanks Cameron, ok so I just ran the exercise again. I killed a server.
> The other two ZKs go offline and the zookeeper.out log is as so:
>
> So why does this feel as if I need to manually adjust the config file?
>
> Thanks. P.S. I am using netflix opensource exhibitor. It may be a factor, I
> don't know. The creator seems to brush me off a lot.
>
> 2015-01-29 00:05:51,271 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
> config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
> tail: zookeeper.out: file truncated
> 2015-01-29 00:06:11,561 [myid:] - INFO [main:QuorumPeerConfig@103] -
> Reading configuration from: /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> 2015-01-29 00:06:11,566 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
> config, exiting abnormally
> org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
> processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
> Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
> invalid configuration
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
> at
>
> org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
> ... 2 more
> Invalid config, exiting abnormally
>
> On Wed, Jan 28, 2015 at 3:45 PM, Cameron McKenzie <mc...@gmail.com>
> wrote:
>
> > Assuming that your 2 remaining servers can communicate with each other
> the
> > cluster should should remain online as it still has a quorum (2 / 3 nodes
> > are online). There may be a brief period where the cluster won't accept
> > requires while a new leadership election occurs between the 2 remaining
> > nodes.
> > cheers
> > Cam
> >
> > On Thu, Jan 29, 2015 at 10:40 AM, Dillian Murphey <
> crackshotmule@gmail.com
> > >
> > wrote:
> >
> > > If I have 3 ZK instances running on separate servers and I terminate
> one
> > of
> > > the servers, what should I expect to happen?
> > >
> > > Will the remaining 2 zookeepers continue happily and stay online
> without
> > > any interruption?
> > >
> > > I'm using exhibitor/zookeeper and I'm attempting this scenerio and what
> > I'm
> > > seeing is my remaining 2 zookeepers go offline. I thought the whole
> > point
> > > of having a majority was to allow a failure?
> > >
> > >
> > > Thank you. I'm new so be kind.
> > >
> >
>
Re: Basic 3 ZK, bring 1 down question.
Posted by Dillian Murphey <cr...@gmail.com>.
Thanks Cameron, ok so I just ran the exercise again. I killed a server.
The other two ZKs go offline and the zookeeper.out log is as so:
So why does this feel as if I need to manually adjust the config file?
Thanks. P.S. I am using netflix opensource exhibitor. It may be a factor, I
don't know. The creator seems to brush me off a lot.
2015-01-29 00:05:51,271 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
invalid configuration
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
... 2 more
Invalid config, exiting abnormally
tail: zookeeper.out: file truncated
2015-01-29 00:06:11,561 [myid:] - INFO [main:QuorumPeerConfig@103] -
Reading configuration from: /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
2015-01-29 00:06:11,566 [myid:] - ERROR [main:QuorumPeerMain@85] - Invalid
config, exiting abnormally
org.apache.zookeeper.server.quorum.QuorumPeerConfig$ConfigException: Error
processing /zookeeper/zookeeper-3.4.6/bin/../conf/zoo.cfg
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:123)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.initializeAndRun(QuorumPeerMain.java:101)
at
org.apache.zookeeper.server.quorum.QuorumPeerMain.main(QuorumPeerMain.java:78)
Caused by: java.lang.IllegalArgumentException: Observers w/o quorum is an
invalid configuration
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parseProperties(QuorumPeerConfig.java:284)
at
org.apache.zookeeper.server.quorum.QuorumPeerConfig.parse(QuorumPeerConfig.java:119)
... 2 more
Invalid config, exiting abnormally
On Wed, Jan 28, 2015 at 3:45 PM, Cameron McKenzie <mc...@gmail.com>
wrote:
> Assuming that your 2 remaining servers can communicate with each other the
> cluster should should remain online as it still has a quorum (2 / 3 nodes
> are online). There may be a brief period where the cluster won't accept
> requires while a new leadership election occurs between the 2 remaining
> nodes.
> cheers
> Cam
>
> On Thu, Jan 29, 2015 at 10:40 AM, Dillian Murphey <crackshotmule@gmail.com
> >
> wrote:
>
> > If I have 3 ZK instances running on separate servers and I terminate one
> of
> > the servers, what should I expect to happen?
> >
> > Will the remaining 2 zookeepers continue happily and stay online without
> > any interruption?
> >
> > I'm using exhibitor/zookeeper and I'm attempting this scenerio and what
> I'm
> > seeing is my remaining 2 zookeepers go offline. I thought the whole
> point
> > of having a majority was to allow a failure?
> >
> >
> > Thank you. I'm new so be kind.
> >
>
Re: Basic 3 ZK, bring 1 down question.
Posted by Cameron McKenzie <mc...@gmail.com>.
Assuming that your 2 remaining servers can communicate with each other the
cluster should should remain online as it still has a quorum (2 / 3 nodes
are online). There may be a brief period where the cluster won't accept
requires while a new leadership election occurs between the 2 remaining
nodes.
cheers
Cam
On Thu, Jan 29, 2015 at 10:40 AM, Dillian Murphey <cr...@gmail.com>
wrote:
> If I have 3 ZK instances running on separate servers and I terminate one of
> the servers, what should I expect to happen?
>
> Will the remaining 2 zookeepers continue happily and stay online without
> any interruption?
>
> I'm using exhibitor/zookeeper and I'm attempting this scenerio and what I'm
> seeing is my remaining 2 zookeepers go offline. I thought the whole point
> of having a majority was to allow a failure?
>
>
> Thank you. I'm new so be kind.
>